Requirements for building NPM packages in distros

[crummel]

A while ago, we added a BuildNodeJs switch to ASP.NET to support source-build using a checked-in version of compiled JS. This was done to remove the dependency on NPM and several NPM packages. ASP.NET has trimmed these dependencies down somewhat and was wondering if they could remove the workaround, which is a bit of extra work for them.

I believe we were only building in two distros at that point and the requirement was determined to be that the NPM packages were also built for the platform and added to the list of build dependencies. I'm still looking at the smaller set of packages to determine if they are all available but we realized that now that we're on more distros we need to account for each distro's policy on this problem. We have a couple questions/discussion points:

• Are NPM packages required to be built and packaged in order to be used as a build dependency in your distro?
• We probably don't need the exact versions listed below; ASP.NET is willing to unify on available versions. If NPM packages are required, are they generally kept up to date?
• Any other comments or concerns you have.

Let us know what you think - I'm going to try to look some of this up as well so I'll updated this thread as I do. Thanks!

cc @dotnet/distro-maintainers

The list of dependencies from ASP.NET:

Blazor:
    "@babel/core": "^7.16.7", 
    "@babel/preset-env": "^7.16.8",
    "@microsoft/dotnet-js-interop": "link:../../JSInterop/Microsoft.JSInterop.JS/src",
    "@microsoft/signalr": "link:../../SignalR/clients/ts/signalr",
    "@microsoft/signalr-protocol-msgpack": "link:../../SignalR/clients/ts/signalr-protocol-msgpack",
    "@types/dotnet": "[./@types/dotnet](mailto:./@types/dotnet)",
    "@types/jsdom": "^16.2.14",
    "@typescript-eslint/eslint-plugin": "^5.26.0",
    "@typescript-eslint/parser": "^5.26.0",
    "eslint": "^8.16.0",
    "eslint-plugin-header": "^3.1.1",
    "inspectpack": "^4.7.1",
    "rimraf": "^3.0.2",
    "terser": "^5.14.2",
    "ts-loader": "^9.2.6",
    "typescript": "^4.5.4",
    "webpack": "^5.72.1",
    "webpack-cli": "^4.9.2"

@microsoft/dotnet-js-interop
    "@typescript-eslint/eslint-plugin": "^5.26.0", 
    "@typescript-eslint/parser": "^5.26.0",
    "eslint": "^8.16.0",
    "eslint-plugin-jsdoc": "^39.3.2",
    "eslint-plugin-prefer-arrow": "^1.2.3",
    "rimraf": "^3.0.2",
    "typescript": "^4.2.2"

@microsoft/signalr
  "devDependencies": { 
    "@types/eventsource": "^1.1.8",
    "@types/jest": "^26.0.20",
    "@types/node": "^14.14.31",
    "@types/tough-cookie": "^4.0.0",
    "process": "^0.11.10"
  },
  "dependencies": {
    "abort-controller": "^3.0.0",
    "eventsource": "^2.0.2",
    "fetch-cookie": "^2.0.3",
    "node-fetch": "^2.6.7",
    "ws": "^7.4.5"

@microsoft/signalr-protocol-msgpack
  "dependencies": { 
    "@microsoft/signalr": "link:../signalr",
    "@msgpack/msgpack": "^2.7.0"
  },
  "devDependencies": {},
  "resolutions": {
    "ws": ">=7.4.6"
  }

[alucryd]

Speaking for Arch Linux, we have very few packaged JS libs in the repository (notable exceptions are some babel packages). Personally, packages I maintain that require JS libs download and ship them with yarn/npm, I don't depend on any repo package.
I have no issue with relying on npm for the dotnet build, the only requirement would be to have a lockfile commited to the repo to have a reproducible build.

[crummel]

Thanks, that's an interesting point. As I understand it that would lock in a specific source hash so maybe that could work as an external source file.

[crummel]

cc @omajid and @mirespace as well

[omajid]

Are NPM packages required to be built and packaged in order to be used as a build dependency in your distro?

The recent policies for building nodejs code in Fedora boil down to:

You can provide a package that uses nodejs, but you should bundle all the nodejs libraries that are needed.

There's even a script nodejs-packaging-bundler to pull down a project from npm.org and all its dependencies for building within a single RPM, offline.

In other words, at least from a Fedora point-of-view, if we knew which exact npm packages and versions were needed (that's what a lockfile does, right?), we should be able to download those npm packages (and their dependencies) and build everything ourselves using the distro-provided node package, fully offline. That would let us be sure we are building everything inside Fedora and but still be able to use pretty much any dependency we need.

That said, I have never directly used this tooling on Fedora side, so I will have to experiment to find out if the bits that are present in the aspnetcore repo are already sufficient for Fedora or not.

[mmitche]

In other words, at least from a Fedora point-of-view, if we knew which exact npm packages and versions were needed (that's what a lockfile does, right?), we should be able to download those npm packages (and their dependencies) and build everything ourselves using the distro-provided node package, fully offline. That would let us be sure we are building everything inside Fedora and but still be able to use pretty much any dependency we need.

THanks @omajid. @steveisok This has applicability to emsdk in SB.

[mirespace]

Hi Chris,

on Ubuntu, we use packages as build dependencies, as avoiding downloading modules using npm/yarn during build time is preferable.

We have a bunch of node-* packages for that (modules already packaged and available for being used in building other packages written in NodeJS). I need to check your list and also the versions (at a quick glance, we don't have node-msgpack, for example, and that will imply creating it in the same way we introduce dotnet in the Archive).

Also, for the internal references as

"@microsoft/dotnet-js-interop": "link:../../JSInterop/Microsoft.JSInterop.JS/src",
"@microsoft/signalr": "link:../../SignalR/clients/ts/signalr",
"@microsoft/signalr-protocol-msgpack": "link:../../SignalR/clients/ts/signalr-protocol-msgpack",
"@types/dotnet": "./@types/dotnet",

ideally (in the debian way) we would need to remove those dependencies code, package them separately and depend on them. But since they already are in the upstream tarball we can make use of them and not download them from the Internet (but I have to admit that I would need to play with that to see how I can do it - That is, if I don't have to rely on the build.sh step to use them - ).

I will provide the list of the modules we have already packaged and the versions matching your shared list.

Thanks Chris!

[crummel]

Hi Miriam, could you clarify on this part?

But since they already are in the upstream tarball we can make use of them and not download them from the Internet

Is that similar to what Omair mentioned where if the packages can be built without internet access it's okay to include them? I also see Debian mentions FPM for turning NPM packages into Deb packages on their wiki but doesn't mention much more - do you have any more details on their policy? Thanks!

[mirespace]

Hi Chris, soooooorry for the delay ... back to work already!

Yes, built without internet it's okay if we can include the necessary into the tarball: I imagine that it will be an extra step (as we did for the arm64 bootstrapping including the nupkg files in packages/prebuilt) and install it as another extra step before start with the build.sh step (but, to be honest, I still have to figure it out... I was looking into how the node debs packages do it using pkg-js-tools ... or going manually :( ... :crossing_fingers:).

About fpm, to my knowledge is a tool that aids you to package/convert node packages to deb ones. I don't know how many packages are submitted to be included in the Archive using this. I also checked (although it seems to me a bad hacking idea), just in case, if any other package use fpm to create intermediate deb packages, like on-the-fly packages, and I didn't find that case.

I still owe you the list of node debs packages!

[ayakael]

Much like Arch, we at Alpine Linux package very few node libraries. We thus allow packaging within our packages node libraries. A lockfile would thus be necessary to build those libraries with source-build.

[omajid]

How to handle npm vs yarn?

I am a node noob and I have just learned that aspnetcore repo uses yarn, not npm. One distribution that I care about - RHEL - has npm, but not yarn. If yarn becomes a hard dependency, that could be an issue for me.

The npm cli is supposed to be compatible for things like pulling down dependencies - it even claims to use the same yarn.lock files.

There are some ways I see to resolve this:

• Some CI leg (in aspnetcore? or vmr?) to verify that npm works as well as yarn when it comes to building?
• Switch to npm?
• yarn is itself a node module too, could we somehow add that to the build dependency graph and download it along with other dependencies?

Any thoughts? Any other possible solutions?

Edit: node -> npm to better reflect the name of the tool

[omajid]

The move from yarn to npm is tracked by dotnet/aspnetcore#37398

[omajid]

Handling the currently-hardcoded Microsoft npm registry?

aspnetcore uses the Microsoft npm registry for downloading node dependencies. I suppose that's because npm.js has rate limits to prevent users from overloading it. It's hardcoded in various .npmrc files in the the projects that use npm/yarn.

However, when I tried building using that registry, I ran into issues because some packages were not present (not sure why) and it needed me to authenticate (before it could delegate? or refresh cache? not sure). Without the authentication tokens, the build failed.

It doesn't seem like a great experience for builders to ask everyone to authenticate to the Microsoft npm registry (even if it was completely open and self-service in that anyone from the community could sign up). Additionally, other organizations may have their own registries that they expect devs to use (at least, Red Hat does). And community devs would probably prefer to use the content from npm.js instead of Microsoft. But we definitely don't want to change the registries used by the current ASP.NET Core/VMR CI systems or devs working at Microsoft.

How do we achieve this?

[alucryd]

I'm pretty sure I'd be hanged if I built a package that downloads stuff from a semi-public authenticated repository, when there's a completely open alternative, even if it's just a mirror. As far as Arch is concerned, these should not be hardcoded but overriden at build time, and use the npm registry by default.