Build process: Adding data file to wheel *after* project is installed locally

[Avasam]

pywin32 used to have a few manual steps as part of building and packaging the project. Which included having pywin32 already installed since it used its own built extensions as part of build scripts.

I managed to solve one of those "catch-22" by migrating a script to pure python (using ctypes) mhammond/pywin32#2349 but recently hit another issue: pywin32 used to ship with a PyWin32.chm help file generated from its source code. Which required running a script manually before the build. And I'd like to restore that functionality as part of a single automated build.

The problem is that the "documentation from Python" script https://github.com/mhammond/pywin32/blob/c717bfa4b06bab5984ba4c9eb62fcc4e6dcf99ef/AutoDuck/pywin32.mak#L118-L125 + https://github.com/mhammond/pywin32/blob/main/AutoDuck/py2d.py uses runtime inspection of imports. Meaning that, once again, pywin32 needs to already be installed to run ! I tried running it as part of a custom install step, but I still need to tell the wheel to pickup that file. I can't add it to data_files because it doesn't exist yet as part of the local install process.

I could attempt to drastically change the doc generation script, hoping that static analysis is sufficient, but before embarking on that.
Is there any way I could:

• build the c extensions
• install locally
• run the doc generation script, with the build extensions imports available
• add the generated file to the wheel

Here's the current setup file: https://github.com/mhammond/pywin32/blob/main/setup.py

This is part of my ongoing effort to modernize pywin32's installation and packaging to be automated from the CI, be fully PEP-517 compliant, support editable installs, and stop using deprecated APIs like distutils or calling setup.py directly.

[Avasam]

Bump. Is this feasible? And how ?

[abravalheri]

Sorry for the delay. This is a highly specialised customisation and probably tricky to achieve.
I wonder if it is something that can be achieved by overloading built_ext.run (and/or build_ext.build_extensions).

At the stage build_ext.run finishes executing, the compiled extensions should be available somewhere, so one could carefully augment sys.path with the appropriate directory so that modules are available for importing. Would that work?

[Avasam]

That might work. I don't mind playing around with the path (pywin32 already does it a lot) to point to the build folder. I'll look into those commands and methods specifically. I think I could update the data_files in run or finalize_options? (Unless this happens before the config-level option is even checked, in which case no dynamic config needed)

Btw an even better use case I plan on for similar customization is to produce stub files from runtime introspection and docstrings of C modules (using mypy's stubgen)

[Avasam]

I couldn't just modify sys.path because it all happens in subprocesses, so I went for creating a .pth file in the site-packages that points ot the build folder, as well as all subfolders pywin32.pth originally pointed to.
It feels super hacky, but it works:
https://github.com/mhammond/pywin32/pull/2432/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7

I'm not sure how/if this will work with build isolation though, once pywin32 adds a pyproject.toml and stops calling setup.py directly. I guess I'll see once I get there. mhammond/pywin32#2396

Edit: Of course I say that, but then it fails in CI context... From the errors I'm getting I know at least the basic idea of pointing a path to the build folder works. There's something else pywin32 does dynamically to find its DLLs that's tripping up and will need investigation...

[abravalheri]

because it all happens in subprocesses

Does it also mean you cannot pass env={**os.environ, "PYTHONPATH": ...} to subprocess.run? (sys.path should take into consideration PYTHONPATH, maybe you need to use something like os.pathsep.join(...) to simply extend whichever PYTHONPATH the parent process is already running with, but probably better than a .pth?)