Port Nix to Meson

[p01arst0rm]

This pull request adds meson buildsystem support. meson allows for better cross platform support. and is a LOT easier to maintain when compared to autotools. There are also some modifications to fix compilation on FreeBSD.

TODO:

• add meson support for tests
• add meson support for docs
• add amazon s3 building

[edef1c]

I'm not sure we're having all that much trouble maintaining the current autotools-based build system, and Nix's build setup is fairly focused around being able to bootstrap it with fairly minimal dependencies. Does meson generate scripts that end up in release tarballs that we can run without having meson available on the build host? I think this needs a lot of justification either way, since we'd be maintaining two build systems rather than one, because I doubt we can drop autotools anytime soon.

[p01arst0rm]

related:

#2503
#1439
#2613

[p01arst0rm]

I'm not sure we're having all that much trouble maintaining the current autotools-based build system, and Nix's build setup is fairly focused around being able to bootstrap it with fairly minimal dependencies. Does meson generate scripts that end up in release tarballs that we can run without having meson available on the build host? I think this needs a lot of justification either way, since we'd be maintaining two build systems rather than one, because I doubt we can drop autotools anytime soon.

the current autotools system leaves a lot to be desired for cross platform support. meson allows a more platform agnostic approach to the buildsystem. Moreover, this port is heavily commented to allow easy future support.

Ninja also compiles significantly faster than autotools/make. It would save developers a substantial amount of time and massively improve productivity.

this adds two dependencies to the source directory; meson & ninja (or muon & samuri). once configured, the build directory only requires ninja/samuri and the previous dependencies. it also removes the dependency on a linux userland

[edef1c]

#2613

Related how? Is this based on @Ericson2314's work, or is this a from-scratch rewrite that just happens to share the same goal?

this adds two dependencies to the source directory; meson and ninja. once configured, the build directory only requires ninja and the previous dependancies.

I'm not sure how to parse this. To my knowledge, ninja doesn't really have conditionals, so if we wanted to produce source tarballs that don't depend on meson we'd have to build ones for specific targets, right?

[p01arst0rm]

#2613

Related how? Is this based on @Ericson2314's work, or is this a from-scratch rewrite that just happens to share the same goal?

This is a from-scratch write, but some parts are similar.

this adds two dependencies to the source directory; meson and ninja. once configured, the build directory only requires ninja and the previous dependancies.

I'm not sure how to parse this. To my knowledge, ninja doesn't really have conditionals, so if we wanted to produce source tarballs that don't depend on meson we'd have to build ones for specific targets, right?

im not quite sure what you're asking either :)

[edef1c]

im not quite sure what you're asking either :)

Trying to tease out what the minimal dependencies for building from a source tarball would end up being, but perhaps that's material for #2503.

[p01arst0rm]

im not quite sure what you're asking either :)

Trying to tease out what the minimal dependencies for building from a source tarball would end up being, but perhaps that's material for #2503.

all dependencies should be labeled inside of the root meson.build (sans meson & ninja)

[Ericson2314]

Yes, this does add python as a build time dep. But I think that's a fine price to pay to get rid of a Unix as a build time dep. (Also, I care more about run time than build time deps as one can always cross compiler from a more normal to a weirder platform.)

[p01arst0rm]

Yes, this does add python as a build time dep. But I think that's a fine price to pay to get rid of a Unix as a build time dep. (Also, I care more about run time than build time deps as one can always cross compiler from a more normal to a weirder platform.)

not to mention cross compilation on meson is considerably easier other platforms

[edolstra]

The current build system has one major advantage: I know how it works. So I can do things like add some Rust code (#2748) without first having to learn how to do that with a completely different build system.

Ninja also compiles significantly faster than autotools/make, to an almost embarrassing extent.

I'm a bit skeptical about this. Some numbers would be helpful. I mean, an incremental "make" in Nix takes 0.05s on my system to determine that it doesn't have to do anything. So the "make" overhead is fairly non-existent.

BTW we don't really have an "autotools-based build system" - we have a Make-base build system. We're not using automake or libtool (anymore). The only autotool we're still using is autoconf and I'd be happy to get rid of it (it doesn't do much that couldn't be done using some Make / C++ conditionals). The approach of detecting rather than specifying platform characteristics is really a remnant of the era when people had to support dozens of obscure Unix systems.

[p01arst0rm]

The current build system has one major advantage: I know how it works. So I can do things like add some Rust code (#2748) without first having to learn how to do that with a completely different build system.
I think it is more helpful to the community to use a system that many people can understand very easily instead of keeping the same system because people are used to it. Meson is VERY simple to understand and has brilliant documentation.

I am also willing to help explain every single line of each script, what it does, why, if/where this is performed in the current build system and why it was included/added.

Ninja also compiles significantly faster than autotools/make, to an almost embarrassing extent.

I'm a bit skeptical about this. Some numbers would be helpful. I mean, an incremental "make" in Nix takes 0.05s on my system to determine that it doesn't have to do anything. So the "make" overhead is fairly non-existent.

I'll run some compilation tests for some numbers shortly ^^

BTW we don't really have an "autotools-based build system" - we have a Make-base build system. We're not using automake or libtool (anymore). The only autotool we're still using is autoconf and I'd be happy to get rid of it (it doesn't do much that couldn't be done using some Make / C++ conditionals).

i believe i said "Ninja also compiles significantly faster than autotools/make". autotools isnt what is causing the awful build performance, make is. switching to ninja takes no overhead on the part of the user, you just type ninja instead of make.

The approach of detecting rather than specifying platform characteristics is really a remnant of the era when people had to support dozens of obscure Unix systems.

The purpose of this re-write is to allow better cross platform support. Just because a make file works on linux, it does not mean it works on minix/*bsd/darwin. The build files for bsd especially have bit rotted to a point where you cannot even compile the source without the modifications included in this PR.

I personally believe Nix should, at least, build on all supported platforms. switching to this meson system will help prevent things breaking in the future. I've also implemented methods to allow a more platform agnostic approach to the build.

[p01arst0rm]

compilation times

meson build
compile start: 1571742622
compile end: 1571742805
total time: 183 seconds

make build
compile start 1571743308
compile end: 1571743716
total time: 408 seconds

[Ericson2314]

@edolstra FWIW meson is is popular among Gnome devs who very interested and C -> Rust things, and so the specific example you mentioned is very much supported.

---

As the killer app for me is removing the unix assumptions without maintain multiple build systems, I'll try to revive https://github.com/volth/nix-windows/ with this during or shortly after NixCon. (I would commit to during and not after if Windows weren't such a pain!)

[zimbatm]

The current build system has one major advantage: I know how it works. So I can do things like add some Rust code (#2748) without first having to learn how to do that with a completely different build system.

How can we make sure that Eelco would be comfortable with Meson? Is anyone willing to be on call for help and provide teaching?

My experience with hacking on Nix is that compilation is quite slow, having faster builds would be a very welcome change to me. Especially since I don't C++ well enough and require more iterations to get anything done :D

[p01arst0rm]

The current build system has one major advantage: I know how it works. So I can do things like add some Rust code (#2748) without first having to learn how to do that with a completely different build system.

How can we make sure that Eelco would be comfortable with Meson? Is anyone willing to be on call for help and provide teaching?

#3160 (comment)

My experience with hacking on Nix is that compilation is quite slow, having faster builds would be a very welcome change to me. Especially since I don't C++ well enough and require more iterations to get anything done :D

same ^^

[edolstra]

@ p01arst0rm Are those compilation times doing the same thing? Namely an -O3 build? And with the same level of parallelism (-j...)?

[p01arst0rm]

@ p01arst0rm Are those compilation times doing the same thing? Namely an -O3 build? And with the same level of parallelism (-j...)?

(performed on different box than the first set)
-j2
-O3

make
compile start: 1572002763
compile end: 1572003137
total time: 374

meson
compile start: 1572003191
compile end: 1572003556
total time: 365

still faster with identical c flags, repeated to check and it was still faster, but Its less dramatic when you set up optimisation flags that way.

update: the make build didnt even finish, i checked the logs and it errored out

[p01arst0rm]

-j4
-O3

make
1572009306
1572009610
304

meson
1572009781
1572010079
298

seems like the difference only matters the more restricted resources are. still faster though.

[Ericson2314]

p01arst0rm#1 I fixed some things here

[nh2]

I would welcome this change.

I think autoconf and in particular m4 is very difficult to contribute to; in my last PR to improve nix's autoconf (#2979, which fixed nix autoconf bugs #257, #506, #967) I spent 2 days to improve it.

In contrast, packages in nixpkgs that use meson to which I recently contributed, like gstreamer, were easier to work with.

I expect that supporting meson will make it way easier for people to contribute to nix and help maintain its build (and prevent that it stops building on e.g. Debian and Ubuntu as happened in the issues mentioned above).

[nh2]

On the idea of using plain make without any configurator:

I would be surprised if it worked well if we want to provide a good workflow for not-in-nix-but-from-source builds, e.g. for other distros. The i3 window manager did for a long time use a plain make file but eventually switched away from it.

Systems like meson find the tools you need (compiler and dependencies, whatever they are named in cross or non-cross cases) by sane means (e.g. pkg-config), allow you to cleanly and unambiguously declare and provide optional dependencies, and give good error messages when something isn't found.

[p01arst0rm]

any updates on this? dont really know if i should keep working on this if its not going to get merged

[Ericson2314]

@p01arst0rm we made the case at NixCon a good deal. Is my PR ready for you to merge? Then I think we just need to fix some Darwin build failures.

[p01arst0rm]

@p01arst0rm we made the case at NixCon a good deal. Is my PR ready for you to merge? Then I think we just need to fix some Darwin build failures.

Theres some things with your PR that arent quite right that im waiting on ^^

[p01arst0rm]

All done :) waiting on review / merge

[p01arst0rm]

any updates on merging?

[Ericson2314]

@p01arst0rm I in part wrote NixOS/rfcs#56 because I think reviewing PRs like this is needlessly difficult without it.

I'll keep on plugging away on the windows stuff in the mean time (NixOS/nixpkgs#73195 was what I last did) and that should bolster the case for this.

Sorry this is not a faster process.

[p01arst0rm]

@p01arst0rm I in part wrote NixOS/rfcs#56 because I think reviewing PRs like this is needlessly difficult without it.

I'll keep on plugging away on the windows stuff in the mean time (NixOS/nixpkgs#73195 was what I last did) and that should bolster the case for this.

Sorry this is not a faster process.

no problem, I just don't want this to bit rot :)

[p01arst0rm]

git history has been corrected, libstore unit tests are now working.

[p01arst0rm]

fixed build for libmain libfetchers libexpr libcmd

[p01arst0rm]

binary build fixed.

[Ericson2314]

@p01arst0rm Now that #10378 is merged, I am curious what you do with this. I want RFC decided on the basis of the nix-perl "trial run" of Meson.

(See also NixOS/rfcs#132 (comment))

[p01arst0rm]

@p01arst0rm Now that #10378 is merged, I am curious what you do with this. I want RFC decided on the basis of the nix-perl "trial run" of Meson.

ill take a break then rebase the pr. hopefully we should be able to figure out porting the documentation without too much hassle :)

[marcusramberg]

These comments does not seem relevant for libgit2, copy pasta?

[p01arst0rm]

looks to be :)

[Ericson2314]

Closing this for now because we've landed PRs to incrementally convert per-project, but I would like to thank @p01arst0rm again for all the hard work with this. It will definitely help us with the remaining conversions too!