remote borg backup to synology nas fails

[soylentOrange]

I wanted to use the a remote borg repository for backing up nextcloud as described in: https://github.com/nextcloud/all-in-one#are-remote-borg-backups-supported
Following the steps in the description (after updating the mastercontainer to the current beta) didn't lead to success, see the log below. The log is complaining that borg isn't available on my remote server (?)

Testing borg on my synology nas turned out successful. ssh-ing to it, I can run borg. Setting up vorta (on my computer) also works as expected.

I'm using ssh on a non-standard port. May it be that the current borg backup solution is only supporting the standard ssh-port 22?

Steps to reproduce

1. installed borg on my synology nas using the SynoCommunity package borg
2. enabled ssh (on a non-standard-port) on the synology nas
3. set up nextcloudAIO to use a remote for back up (ssh://[email protected]:YYYY/volume1/BorgBackup/nextcloud_backup)
4. started an initial backup to generate the ssh public key
5. added the ssh public key to authorized keys on the synology nas
6. initiated a backup again --> that failed (see log below)

Expected behavior

A backup should be created

Actual behavior

initiating a backup after setting up the borg backup fails (see log below)

Other information

Host OS

AIO is installed on a synology nas (DS720+ with DSM 7.2.2-72806 Update 1)

Output of sudo docker info

Client:
Context: default
Debug Mode: false

Server:
Containers: 10
Running: 2
Paused: 0
Stopped: 8
Images: 12
Server Version: 20.10.23
Storage Driver: btrfs
Build Version: Btrfs v4.0
Library Version: 101
Logging Driver: db
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs db fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b23a389d8c181697302d163356e97dec04eb8d88
runc version: 5af893d
init version: ed96d00
Security Options:
apparmor
Kernel Version: 4.4.302+
Operating System: Synology NAS

OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.712GiB
Name: biomech-lab
ID: PE5P:7Q7V:FCVG:3QFA:SFXH:4GUY:IZS2:MHU3:QIXG:FYSC:XOLP:5JZJ
Docker Root Dir: /volume1/@docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No kernel memory TCP limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Docker run command or docker-compose file that you used

docker run -d --name=nextcloud-aio-mastercontainer
-p 8080:8080
-v /var/run/docker.sock:/var/run/docker.sock:ro
-v nextcloud_aio_mastercontainer:/mnt/docker-aio-config
-e APACHE_PORT=11000
-e APACHE_IP_BINDING=0.0.0.0
-e NEXTCLOUD_UPLOAD_LIMIT=10G
-e NEXTCLOUD_MEMORY_LIMIT=4096M
-e COLLABORA_SECCOMP_DISABLED=true
-e SKIP_DOMAIN_VALIDATION=true
--restart always
--init
--sig-proxy=false
nextcloud/all-in-one:beta

Other valuable info

Backup log:

2024-11-18T09:05:12.511576554Z Your public ssh key for borgbackup is: XXXXXX
2024-11-18T09:05:14.212664835Z Remote: sh: borg: command not found
2024-11-18T09:05:14.220577674Z Connection closed by remote host. Is borg working on the server?
2024-11-18T09:05:14.284188251Z Initializing repository...
2024-11-18T09:05:14.946487680Z using builtin fallback logging configuration
2024-11-18T09:05:15.351710673Z 33 self tests completed in 0.40 seconds
2024-11-18T09:05:15.353697126Z SSH command line: ['ssh', '-o', 'StrictHostKeyChecking=accept-new', '-i', '/nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/id_borg', '-p', 'YYYY', '[email protected]', 'borg', 'serve', '--debug']
2024-11-18T09:05:15.551342559Z Remote: sh: borg: command not found
2024-11-18T09:05:15.558443248Z Connection closed by remote host. Is borg working on the server?
2024-11-18T09:05:15.558575277Z Traceback (most recent call last):
2024-11-18T09:05:15.558613380Z File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 5401, in main
2024-11-18T09:05:15.558671482Z exit_code = archiver.run(args)
2024-11-18T09:05:15.558702076Z ^^^^^^^^^^^^^^^^^^
2024-11-18T09:05:15.558727047Z File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 5321, in run
2024-11-18T09:05:15.558758091Z return set_ec(func(args))
2024-11-18T09:05:15.558789400Z ^^^^^^^^^^
2024-11-18T09:05:15.558815812Z File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 168, in wrapper
2024-11-18T09:05:15.558849166Z repository = RemoteRepository(location.omit_archive(), create=create, exclusive=argument(args, exclusive),
2024-11-18T09:05:15.558884885Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-18T09:05:15.558914407Z File "/usr/lib/python3.12/site-packages/borg/remote.py", line 578, in init
2024-11-18T09:05:15.558945528Z raise ConnectionClosedWithHint('Is borg working on the server?') from None
2024-11-18T09:05:15.558976749Z borg.remote.ConnectionClosedWithHint: Connection closed by remote host. Is borg working on the server?
2024-11-18T09:05:15.559007340Z
2024-11-18T09:05:15.559031198Z Platform: Linux 636cd313dc01 4.4.302+ #72806 SMP Thu Sep 5 13:44:44 CST 2024 x86_64
2024-11-18T09:05:15.559080298Z Linux: Unknown Linux
2024-11-18T09:05:15.559108326Z Borg: 1.2.8 Python: CPython 3.12.7 msgpack: 1.0.8 fuse: llfuse 1.5.0 [pyfuse3,llfuse]
2024-11-18T09:05:15.559143840Z PID: 35 CWD: /
2024-11-18T09:05:15.559170505Z sys.argv: ['/usr/bin/borg', 'init', '--debug', '--encryption=repokey-blake2']
2024-11-18T09:05:15.559202246Z SSH_ORIGINAL_COMMAND: None
2024-11-18T09:05:15.559228744Z
2024-11-18T09:05:15.622236354Z Could not initialize borg repository.

[szaimen]

cc @timdiels

[soylentOrange]

OK, I think I found the source of the problem and a workaround.
When installing borg from the SynoCommunity package, it populates /usr/local/bin/borg and not /usr/bin/borg.
When ssh-ing from my computer to the synology nas with: ssh [email protected] -p YYYY -i ~/.ssh/key borg it fails with the very same error as in the logs above (sh: borg: command not found).

After creating a symlink using the task planner of the synology nas, run as root with a user defined script ln -s /usr/local/bin/borg /usr/bin/borg , the borg backup from nextcloudAIO starts working successfully.

So, it's not really a bug of nextcloudAIO but just a synology related problem...