Mirror all JDownloader repositories

[darkdragon-001]

Please also mirror dependency repositories:

• AppWork Utilities: svn://svn.appwork.org/utils
• JDBrowser: svn://svn.jdownloader.org/jdownloader/browser
• MyJDownloader: svn://svn.jdownloader.org/jdownloader/MyJDownloaderClient

[T3rm1]

Why do they even still use SVN?

[milahu]

• svn://svn.jdownloader.org/jdownloader
  • https://github.com/mycodedoesnotcompile2/jdownloader_mirror
  • https://github.com/mirror/jdownloader - 2023, outdated
  • https://sourceforge.net/p/jdownloader-svn-mirror-milahu/code/
    • pull: rsync --recursive --info=progress2 svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader/
      • fast: 90 rev/sec. rsync is 100x faster than svn → fast initial sync, then slow update from upstream svn repo
    • push: svnsync sync svn+ssh://[email protected]/p/jdownloader-svn-mirror-milahu/code/ file://$PWD/jdownloader/
      • pushing requires hooks/pre-revprop-change - see SVN Import
      • slow: 0.4 rev/sec
• svn://svn.appwork.org/utils
  • https://github.com/milahu/appwork-utils
  • https://sourceforge.net/p/milahu-appwork-utils/code/
    • pull: rsync --recursive --info=progress2 svn.code.sf.net::p/milahu-appwork-utils/code/ appwork-utils/
    • mirror was created with svnsync. i dont know why my username (milahu) is used as commit author...
• svn://svn.jdownloader.org/jdownloader/browser
  • part of the main jdownloader repo
• svn://svn.jdownloader.org/jdownloader/MyJDownloaderClient
  • part of the main jdownloader repo

sync tool https://github.com/milahu/svn2github - requires github ssh public key

#!/bin/bash
githubUser=milahu
set -e # fail on error
for repoName in appwork-utils jdownloader-browser jdownloader-client
do
  svn2github update $githubUser/$repoName
done

[milahu]

Please also mirror dependency repositories

this also makes sense, because
svn://svn.jdownloader.org/jdownloader is a monorepo with 45K revisions
and if you do a partial mirror, the revisions are not continuous (aka "sparse")

the jdownloader documentation says we should add the folders separately
but most commits are in trunk/ = jdownloader sources
so there is only little overhead in using the monorepo

svn2github

i tried to mirror the full repo to github, but github says

[GitHub] Git LFS disabled for milahu

Git LFS has been disabled on your personal account milahu because you’ve exceeded your data plan by at least 150%. Please purchase additional data packs to cover your bandwidth and storage usage:

https://github.com/account/billing/data/upgrade

Current usage as of 12 May 2022 02:50AM UTC:

Bandwidth: 0.0 GB / 1 GB (0%)

Storage: 1.6 GB / 1 GB (160%)

i guess 1.6GB means the bare git repo size (du -sh .git/)

anyway, i gave up on mirroring this to github

but i still have a mirror on sourceforge:
https://sourceforge.net/p/jdownloader-svn-mirror-milahu/code/HEAD/tree/
this can be cloned with rsync in about 8 minutes

rsync -r rsync://svn.code.sf.net/p/jdownloader-svn-mirror-milahu/code/ ./jdownloader --verbose

problem is, you cannot verify the integrity of my mirror
so to make sure, you still need to fetch the original svn repo

svnsync of the original repo takes 6 hours (plot)
svnrdump should be much faster

svnrdump dump --quiet --incremental --revision 0000:0999 --file svn-dump.0 svn://svn.jdownloader.org/jdownloader
svnrdump dump --quiet --incremental --revision 1000:1999 --file svn-dump.1 svn://svn.jdownloader.org/jdownloader
svnrdump dump --quiet --incremental --revision 2000:2999 --file svn-dump.2 svn://svn.jdownloader.org/jdownloader
# ...

to convert svn to git, i use svn-all-fast-export
which is much faster than the old svn2github implementation via git svn fetch

i wrapped everything into a python script svn2github which is not perfect, but it's a start

happy mirroring ; )