[WIP] dymamic crossenv + python3*-wheels + python310-313 updates #6282
Conversation
|
@hgy59 I'll let this run but this patch in particular 9c764a7 needs to be in its own PR... interesting from not having noticed it earlier. I'll also mark as [WIP] as not even close to be merged, this was early work being push to initiate exchange with python github relatively to the build issue I had. |
th0ma7
changed the title
|
@hgy59 |
@th0ma7 I will remove this in #6269. It is obsolete since spksrc image on debian 12. |
|
@hgy59, @mreid-tt and other @SynoCommunity/developers I'd appreciate having some thoughts on this... I always kept within Doing so I thought why not building them all and include other remanants dispersed into other sub
Then I recall someone mentioning a while back ago, why not having our own wheel repository?
Thoughts on this would be much appreciated. Also on my TODO (to which babysteps may be best):
|
|
@th0ma7, I’m not very experienced with Python builds, so I had to familiarize myself with the basics (e.g., using Real Python's guide on wheels). Here are some initial thoughts:
I might be missing some key points due to my limited background, so I'd appreciate more details. For instance, you mentioned integrating other "remnants" scattered across sub SPKs. What exactly are these remnants, and what benefits would centralizing them bring? Are they also related to test code? Regarding the internal wheel repository, is it a matter of PyPI's offerings being insufficient? Are there commonly missing platform-specific wheels for Synology hardware? Would it be feasible to advocate for the Python Package Index to include the wheels we need, perhaps by reaching out to the relevant projects and requesting support? I might not fully grasp the situation, but I’m keen to understand the challenges better and contribute more meaningful suggestions. |
Indeed. Package could even be renamed similarly.
Not really. That would mostly be just a web page listing our wheels, that you can query through
Issue is, pypi provides tons of pre-compiled wheels but ppc, armv5, (and often armv7 and even aarch64) are always missing. Therefore, on our NAS at installation time Therefore when we build our spk package using our spksrc framework, we pull source packages from pypi usng
The challenge is that the python wheel build system is undergoing a lot of changes currently and impacting the overall wheel building approaches. As pypi modules documentation and maintenance varies from one to the next, this ends-up breaking things all over the place. So question is, would it be easier to have a single location to manage all wheel cross-compiling, providing numerous versions online to ease package management? I've been thinking of this a little more and still unsure, further as this would probably mean statically linking wheels so it is compatible with any installation OR using rpath so dependencies are available from that |
|
It makes sense to have our own repository. Although it will be quite a lot of work to implement and significantly add to maintenance burden.. |
th0ma7
force-pushed
the
python-wheels
branch
from
bb787f5 to
a7a0997
Compare
|
@hgy59 I believe I may now have a feature-wise functional code ready for testing... I'll be away next week (SC24) so cycles until my return may be limited. Although I would much like your opinion on this invasive but in theory fully backward compatible code change. TL;DR;
Nice addition (from my perspective), ability to re-generate crossenv on demand (see description above for howto). Besides the non-blocker TODO items, in theory it should be ready for testing and shaking out bugs and/or adjusting proposed strategy. I'm mostly thinking of weird corner-cases, such as the ones found in homeassistant that I hope can be addressed with this (to be tested + wheel specific crossenv configurations to be created). Let me know if you have a moment to test-bed this, your pair of 👀 would be much appreciated. Lastly: I did not forgot you with your requirement to join ffmpeg + wheel cross-compiling. With the previous addition of EDIT: Looking at github-action output it seems there are still a few rough edges to look for... on my todo list. |
|
initial analysis: when building python311 for the first time, the when you call make again, the but after call the the variables are set (and build finally succeeds): |
|
some more details to the analysis above all lines from build log starting with This shows that
the first has correct variables, but the second is missing the python version This might trigger you... |
|
@th0ma7 another idea I want to share When wheels couldn't be built without additional wheels in crossenv (like expandvars to build frozenlist), I was looking in the |
That would be elegant indeed, and this pr could set the stage as a start to move towards gainin more flexibility with wheel building. |
|
@hgy59 now fixed and ready for testing. I also migrated the code to make use of status cookie handling like other pieces of the framework which simplify things a lot. If by any chances you have a moment to look at py313 cross-compiling... they moved away from |
|
not yet testing... it fails to install For DSM6.2.4 it fails to install We must either
List of additional modules installed into crossenv (those are not listed in |
|
@hgy59 question of managing expectations... This PR really still is a I was hoping that at this stage it could be tested to confirm wetter this suffice to resolve the immediate build failures we have and allow getting reproducibility back for some of our packages (
Interestingly it's only python 3.13 that fails, and I'm glad it only fails on this as I had other issues previously which now looks solved, at least thru github-action.
The
Currently testing that to see if this helps... Also, on my local branch cross/python313 currenly fails with the following but builds fine using I'll have another look at it upon my return (feel free to push fixes if you hapen to have cycles).
I tried this... and it's really tricky as ordering of install is important. On the other hand that may ease passing But you're right, while downgrading
yup, due to the dependency chain as we need to provide all dependencies for cross-compiling to actually work. I find it handy to print the list of crossenv installed wheels to track exactly what is the build environment in use when cross-compiling. Lastly, my next step is to review the wheel building code to use status cookies and be closer to the remaining of the framework code. I want to divide it such as the following for a start:
So then it will become much easier to maintain and add extra functionalities as needed such as meson and cmake toolchain file support, and potentially automating |
|
Qoriq failure may be related to python/cpython#125269 |
|
@hgy59 lzma hopefuly fixed now. Somehow it isn't able to find that library from being symlinked to the actual |
th0ma7
changed the title
|
@th0ma7 I am trying to create wheels for newer homeassistant with python312. To create unfortunately maturin does not see the installed typing_extensions is there something missing? |
|
Oh that is really tricky... It actually uses maturin from the crossenv in terms of dependencies when building the wheels... But maturin per say really is being used from the native python build. I never found a way to have maturin available part of the crossenv similarly to cross-pip. Admittedly it worked so i left it as is... |
|
@hgy59 this got me thinking... and reminder, I'm working from prior-art and I don't have the full knowledge on this, but I think I found out the obvious from something that got ported from one python migration to the next in a form or another which is now only in I believe that this needs to be per-crossenv specific and be referring to Because, if you look at the crossenv:
The I doubt I'll have time to confirm my theory before the weekend but I thought I'd let you know in case you do have a few spare cycles. |
|
@hgy59 I was too curious and had a few cycles to test something up. I believe this should solve your issue. |
but it breaks the build of spk/python*** so the path is required to build python. |
- --with-system-ffi is not supported (python312, python313) - --with-system-ffi is "is ignored on this platform" (python311)
- python310 and python311 require the path to build python for "generate-posix-vars"
- add wheelhouse to package - many wheels need -std=c11 for gcc4 - llfuse wheel fails to build (temp. excluded) - remove ARMv5 specific handling - udpate rpds_py==0.20.0 and remove requirements-crossenv-rpds-py.txt (not referenced) - add specific crossenv for pydantic_core
|
@th0ma7 there are still some issued (even I fixed some)
|
|
@th0ma7 remarks for the I removed the code that defined I already had this changes prepared in the work on homeassistant for py312, and needed to verify only for ARMv7L that is not supported by ha. |
…rossenv - use python version specific crossenv - downgrade cryptography in default crossenv for python < 3.12 to v41.0.3
|
@th0ma7 the latest changes have beaten me... the build of spk/python*-wheel (and probably other python-module packages too) do not work anymore. |
|
Not surprising as this is really tricky work. Give me a few days so i can reconvene and look into that, this week was a busier one and cycles have been scarce at best. |
some thoughts:
|
|
deluge fails buiding libtorrent wheel, not finding |
|
@hgy59 was overwhelmed with other priorities, should be able to reconvene this week, will keep you posted. cheers! |
Description
Intent is to:
python310/python311baseMakefilesto ease updatespython312andpython313packagescrossenvcreation frompython3*/Makefile-> Now usesspksrc.crossenv.mkcrossenvenablement using amk/crossenv/directory containing wheel crossenv definitions:Fixes #6284
Checklist
all-supportedcompleted successfullyType of change
TODO
$HOME/.cache/pipto use$(WORK_DIR)/piprequirements.txtentry with trailing commenttest==1234 # This is a test wheel$(HOME)/.cache/pipOPENSSL_*_DIRvariables and logic usage throughout python related mk filescrossenvadd status info tostatus-build.logfilespksrc.python-wheel.mkto use status cookie to avoid always rebuilding -->> LEFT FOR SUBSEQUENT PRzlibcreation + remove of symlinks from spksrc.python.mk to eliminate rebuilding