1.19.0 compressed tarball > 10MB bigger than 1.18.0

[rofl0r]

Description

the bz2 tarball snapshot for 1.19.0 from libretro's gitlab is > 10 MB bigger than the one for 1.18.0, the .gz available from github even more.

[Description of the bug]

the main culprit seems to be the addition of binaries
https://github.com/libretro/RetroArch/blob/00b8a5f15c7d671cdf107c3d01e71a729581d1cd/pkg/apple/Frameworks/MoltenVK.xcframework/ios-arm64/MoltenVK.framework/MoltenVK
from pr #16463 by @warmenhoven . i doubt these binaries have any legitimate reason to not only bloat the repo itself, but also source tarballs.
they should be removed and fetched from somewhere else for people doing ios aarch64 builds (or the CI), if they cant be easily be built from source.
the tarball of 1.18.0, weighing at >50 MB is already way more than big enough.

[warmenhoven]

It's a fair point and something I'm not happy with either, but tldr, I'm not planning on doing anything about it.

As a reference point, the 1.19.0 tgz file is approximately 64MB. A tgz file containing only MoltenVK.framework is 18MB. So, it is quite significant, but the tgz size is increased by less than 30%.

There are a few options for how to bring MoltenVK in, but basically the permutations are: source or binary, bundled or external, and then potentially different decisions for repo vs tarball.

Distributing source is not a great option: Building MoltenVK from source is mostly straightforward but very time consuming; moreover the source distribution for MoltenVK and its dependencies is roughly two orders of magnitude greater than 18MB compressed. That is complicated by the fact that our MoltenVK framework is actually two different versions of MoltenVK; we use an older version on tvOS because of incompatibilities in newer versions of MoltenVK with old versions of tvOS. Also we have custom patches on top of MoltenVK (to provide HDR support on iOS and tvOS using the pixel format we use) so just expecting the user to provide them will not work well. (Of course the source for MoltenVK is available for those who want it, but separately.)

For the git repo, keeping the binary in the repo is again not great but adds a relatively insignificant overhead to the 738MB for a fresh clone. The binaries do not change regularly. Having the build system fetch them has caused problems for the CI system for similar related fetches in the past. And having them immediately available for people who just want to build RetroArch for macOS/iOS/tvOS keeps the build instructions simple (right now you only need to install Xcode, clone the source, and open the Xcode workspace; there's no confusion about how to install them so Xcode can find them). The lower barrier to entry is worthwhile in my opinion. (I'll note that other popular emulators like ppsspp have taken this approach as well.)

For the tarball, this one I am a little bit more ambivalent to. It would be possible to have two tgz files: one without MoltenVK entirely, and the other either just MoltenVK or the combination of the two. I'm of the opinion that the simplicity of only having one source distribution that is complete for all platforms is beneficial and worth the cost of having that one tgz file be 30% bigger. Clearly there's a limit at which that conclusion changes and I understand each person might set that limit differently, but I'm comfortable with it as is and don't intend to address it.

As a footnote, the increase from 1.18 to 1.19 was only 10MB even though MoltenVK is more than 18MB; this is because between the two versions, support for compiling for the iOS and tvOS simulators was added. The simulators are an invaluable development and debugging tool and iOS developers would naturally expect that the source distribution should work for the simulator. I don't consider any of the five platforms (macOS, iOS, iOS simulator, tvOS, tvOS simulator) less valuable or worthy of being included in the source distribution.

[rofl0r]

in my opinion, binaries have no place in source code repositories, especially if they're used during the build - as in executed or linked in. this is quite contrary to the expectation of people trying to build such source code. they will probably not even be aware of having some preexisting binary code linked in or executed on their system - which could be malicious (mind you, i'm not suggesting you're deliberately planting malicious code, but the system you've been using to build those binaries could already have been infected).

aside from the trust aspect, the size is the biggest issue. once in the git history, you'll never be able to remove them, unless you rewrite the history. even if the binaries are not expected to change often, they will have to at some point.
as you mentioned, the current clone is already >700mb, most likely because some people checked in binaries in the past. if care had been taken to avoid this from the beginning, the complete clone would almost certainly not exceed 50mb.

we had this problem with the openbor svn repo, where a simple checkout was over 2gb because they had checked in complete toolchains, 7zip installers for windows and practically everything a windows user would need to compile the repo for any supported platform. having to fetch 2gb from a slow network is quite a high barrier to entry for a new contributor who just wants to contribute a patch.

ppsspp is another case where an outrageous amount of data has to be fetched to compile from source, split overall several git submodules, including lots of prebuilt library code.

now let's look at possible solutions:

• the ci or whoever needs the prebuilt moltenvk could simply fetch the existing 1.19.0 tarball which already has it included and extract that part into the source tree.
• the prebuilt stuff could be placed in a git submodule. the CI or user would simply need to do a recursive clone. (but please don't use submodules for code, else the source distribution becomes a mess). anyone else can just clone without submodules and still get the complete code.
• if you really insist on keeping the git repo cluttered with binaries, a CI job that's executed on pushing a tag could create a source tarball with binaries deleted before packing it up and uploading the artifact. this could even produce a tar.xz which has way better compression than both gz and bz2 while extracting much faster than bz2 and only slightly slower than gz.

i submitted such a CI job to ppsspp, because it was impossible to get a complete source snapshot without using a recursive git clone. the job is launched after creating a release, does a recursive git clone, then removes all known pre-built binaries and creates a tar.xz.
this allows distro maintainers to use existing infracstructure to download the entire release in a single moderately sized (about 40mb) source tarball, without being forced to clone about 2 gigs of submodules.

[rofl0r]

as a test, i removed pkg wii deps/glslang/glslang/Test which are all superfluous for a POSIX platform, and the resulting tar.bz2 is 14 mb, and as tar.xz 11mb. that seems to be a reasonable size for distributing the source for POSIX platforms (linux, BSD, haiku, etc).
would there be interest in having a CI job that creates such a tarball on tagging a release ?
i could probably whip that up in half an hour, given that i already did almost the same thing for ppsspp. the tarball could be named e.g. "retroarch-1.19.2-posix-source.tar.xz"
(though i personally think that deps/glslang/glslang/Test should be removed entirely from the repo, "wii" - which contains pre-built libraries for wii be moved into pkg, and pkg be turned into a subrepo)
@LibretroAdmin @warmenhoven

[schellingb]

I agree with the sentiment, purely because I think one of the great appeals of RetroArch for new contributors is the ease of entry. I wouldn't be here weren't it for the relatively small size of the repository and griffin.c making it easy as pie to compile and run RetroArch without needing to trust some convoluted opaque build system I have never heard of.

I like to keep things light, so back when github supported SVN as clients I used sparse checkout to remove the /pkg directory and a few more stuff not relevant to the platform I'm working on. Now that I'm forced to use git, I limited the amount of history cloned with --depth and then I deleted pkg and marked the deletions locally as "assume-unchanged" so git ignores it. Though the data still exists in .git/objects, can't get rid of that.

# to ignore locally deleted files after cloning the repo and deleting the uninteresting files/directories:
git ls-files --deleted -z | git update-index --assume-unchanged -z --stdin

Here's some big stuff in the RetroArch repo (of the total 227 MB ) not every developer needs:

• 60.8MB: /pkg/apple/Frameworks/MoltenVK.xcframework
• 14.1MB: /pkg/apple/ (remaining stuff besides MoltenVK)
• 20.2MB: /pkg/msvc-uwp
• 7.93MB: /pkg/windows
• 10.3MB: /pkg/ctr
• 5.42MB: /wii/libogc
• 11.3MB: /deps/SPIRV-Cross/reference (is this even needed? Looks like documentation on SPIRV?)
• 11.5MB: /gfx/include/userland (no idea what this is, there seems to be a 3MB "hello_pi" app in there? https://github.com/search?q=repo%3Alibretro%2FRetroArch+userland+NOT+path%3Agfx%2Finclude%2Fuserland&type=code doesn't list much)

Unused things like the large samples and documentation from SPIRV or userland should be wiped I think. The MoltenVK framework binaries for Apple platforms sure stand out. Would it somehow be possible to have them as source under /deps/ and compile it as needed? The source is probably also humongous but maybe smaller than binaries for 5 separate platforms.

[zoltanvb]

(Userland is for Raspberry Pi, see #15972, that is awaiting testing from someone who can check CRTSwitchRes operation)