builder: defer submodule detection to git

[abn]

Recent builds on copr started failing for rpms using submodule aware builders, see logs below. The root cause for the issue was the presence of an empty .gitmodules file in a submodule. The recursion logic currently only checks if the file exists, if it does, tries to fetch the paths using the git config --file .gitmodules --get-regexp path command. Which exits with status code 1 when the .gitmodules file is empty.

I have tried to work around this issue by deferring the logic of identifying all nested submodules to git submodule status --recursive instead. Which allows for a less complex handling within tito. Ideally, I reckon this could even be simplified further by use of either the git submodules foreach 'git archive ...' command or git ls-files --recurse-submodules and piping those files into the tarfile module. But for now, I have kept the change trivial and attempted to keep the interface change to a minimum.

Please let me know if I need to fix/add anything.

copr-build.log

Running: tito build --srpm --output /var/lib/copr-rpmbuild/results

cmd: ['tito', 'build', '--srpm', '--output', '/var/lib/copr-rpmbuild/results']
cwd: /var/lib/copr-rpmbuild/workspace/workdir-lc93u41j/web-eid-rpm
rc: 1
stdout: Creating output directory: /var/lib/copr-rpmbuild/results
Checking for tag [web-eid-2.6.0-1] in git repo [https://github.com/abn/web-eid-rpm.git]
Building package [web-eid-2.6.0-1]
stderr: ERROR: Error running command: git config --file .gitmodules --get-regexp path

Status code: 1

Command output: 

Traceback (most recent call last):
  File "/usr/bin/tito", line 33, in <module>
    sys.exit(load_entry_point('tito==0.6.26', 'console_scripts', 'tito')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/tito/cli.py", line 944, in main
    CLI().main(sys.argv[1:])
  File "/usr/lib/python3.12/site-packages/tito/cli.py", line 210, in main
    return module.main(argv)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/tito/cli.py", line 432, in main
    return builder.run(self.options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/tito/builder/main.py", line 165, in run
    self.srpm()
  File "/usr/lib/python3.12/site-packages/tito/builder/main.py", line 224, in srpm
    self.tgz()
  File "/usr/lib/python3.12/site-packages/tito/builder/main.py", line 578, in tgz
    self._setup_sources()
  File "/usr/lib/python3.12/site-packages/tito/builder/submodule_aware_builder.py", line 61, in _setup_sources
    self.create_tgz(
  File "/usr/lib/python3.12/site-packages/tito/builder/submodule_aware_builder.py", line 192, in create_tgz
    for submodule_tar_file in self._submodule_archives(
  File "/usr/lib/python3.12/site-packages/tito/builder/submodule_aware_builder.py", line 155, in _submodule_archives
    for archive in self._submodule_archives(
  File "/usr/lib/python3.12/site-packages/tito/builder/submodule_aware_builder.py", line 155, in _submodule_archives
    for archive in self._submodule_archives(
  File "/usr/lib/python3.12/site-packages/tito/builder/submodule_aware_builder.py", line 117, in _submodule_archives
    submodules_output = run_command(submodules_cmd)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/tito/common.py", line 305, in run_command
    raise RunCommandException(command, status, output)
tito.exception.RunCommandException: Error running command: git config --file .gitmodules --get-regexp path

Copr build error: ERROR: Error running command: git config --file .gitmodules --get-regexp path

Status code: 1

Command output: 

Traceback (most recent call last):
  File "/usr/bin/tito", line 33, in <module>
    sys.exit(load_entry_point('tito==0.6.26', 'console_scripts', 'tito')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/tito/cli.py", line 944, in main
    CLI().main(sys.argv[1:])
  File "/usr/lib/python3.12/site-packages/tito/cli.py", line 210, in main
    return module.main(argv)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/tito/cli.py", line 432, in main
    return builder.run(self.options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/tito/builder/main.py", line 165, in run
    self.srpm()
  File "/usr/lib/python3.12/site-packages/tito/builder/main.py", line 224, in srpm
    self.tgz()
  File "/usr/lib/python3.12/site-packages/tito/builder/main.py", line 578, in tgz
    self._setup_sources()
  File "/usr/lib/python3.12/site-packages/tito/builder/submodule_aware_builder.py", line 61, in _setup_sources
    self.create_tgz(
  File "/usr/lib/python3.12/site-packages/tito/builder/submodule_aware_builder.py", line 192, in create_tgz
    for submodule_tar_file in self._submodule_archives(
  File "/usr/lib/python3.12/site-packages/tito/builder/submodule_aware_builder.py", line 155, in _submodule_archives
    for archive in self._submodule_archives(
  File "/usr/lib/python3.12/site-packages/tito/builder/submodule_aware_builder.py", line 155, in _submodule_archives
    for archive in self._submodule_archives(
  File "/usr/lib/python3.12/site-packages/tito/builder/submodule_aware_builder.py", line 117, in _submodule_archives
    submodules_output = run_command(submodules_cmd)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/tito/common.py", line 305, in run_command
    raise RunCommandException(command, status, output)
tito.exception.RunCommandException: Error running command: git config --file .gitmodules --get-regexp path

[abn]

If anyone has a workaround for this without waiting for a new release, I would appreciate the pointers.

[FrostyX]

Hello @abn,
thank you for the PR.

I don't use this builder in any of my projects. @arcivanov, @t0fik could you please try the PR to make sure it doesn't break your use-cases?

[abn]

Addressed the linting errors reported by last ci run.

[abn]

@FrostyX sorry had to re-push since I figured the linting fixes are better off in a separate commit.

[t0fik]

If it does not reintroduce #414 and #413. It's fine for me.

[arcivanov]

I need a little more time to confirm it works, sorry.

[arcivanov]

LGTM

[abn]

Looks like the PyLint rules are in conflict with the epel-7 build environments.

Bytecompiling .py files below /builddir/build/BUILDROOT/tito-0.6.26-1.20241106135247247702.pr507.18.g350afb0.el7_9.x86_64/usr/lib/python2.7 using /usr/bin/python2.7
Compiling /builddir/build/BUILDROOT/tito-0.6.26-1.20241106135247247702.pr507.18.g350afb0.el7_9.x86_64/usr/lib/python2.7/site-packages/tito/builder/submodule_aware_builder.py ...
  File "/usr/lib/python2.7/site-packages/tito/builder/submodule_aware_builder.py", line 122
    submodule_tar_file = f"{initial_tar}.{submodule_tar_file_suffix}"

https://download.copr.fedorainfracloud.org/results/packit/rpm-software-management-tito-507/epel-7-x86_64/08221794-tito/builder-live.log.gz

The fix was introduced due to failures in PyLint checks in the original workflow runs. Should I maintain python2 compatibility?

[FrostyX]

Thank you very much @t0fik and @arcivanov for testing.
In that case, it is +1 from me as well.

The fix was introduced due to failures in PyLint checks in the original workflow runs. Should I maintain python2 compatibility?

There was a discussion about dropping RHEL7 in a recent PR. I am not opposed to doing so, especially if it complicates the development/maintenance. But here it seems we can just replace f-string with "{0}.{1}".format(initial_tar, submodule_tar_file_suffix)

Can you please do this one last thing @abn and then I will merge and publish a new release with your fix.

[abn]

@FrostyX I have replaced all the lint fixes with the format you have suggested

[FrostyX]

Thank you very much :-)