gateway: unable to create lockfile when running as service

[sedlund]

What happened:
Using user:group juicefs:juicefs to start gateway. Get error trying to create lock. The service starts, but continues to complain that it cannot create the lock file. Which may lead to data corruption?

Dec 01 13:57:44 sgp1 systemd[1]: Started JuiceFS S3 Gateway.
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.462108 juicefs[2545041] <INFO>: Meta address: redis://sgp1.jackal-godzilla.ts.net:6379 [interface.go:504]
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.468388 juicefs[2545041] <INFO>: Ping redis latency: 537.065µs [redis.go:3515]
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.471914 juicefs[2545041] <INFO>: Data use s3://sedlund-jfs/sedlund-jfs/(encrypted) [gateway.go:230]
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.472696 juicefs[2545041] <INFO>: Disk cache (/var/cache/juicefs/5cf44ea1-3e99-4ead-af09-029d332c28f5/): capacity (100 GiB), free ratio 10%, used ratio - [space 59.9%, inode NaN%], max pending pages 15 [disk_cache.go:144]
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.476417 juicefs[2545041] <INFO>: Create session 321 OK with version: 1.2.1+2024-08-30.cd871d19ba57 [base.go:398]
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.477769 juicefs[2545041] <INFO>: Prometheus metrics listening on 127.0.0.1:9567 [mount.go:135]
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.595852 juicefs[2545041] <ERROR>: failed to open the file to be locked: .minio.sys/config_lock error permission denied [gateway.go:1300]
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.595969 juicefs[2545041] <WARNING>: failed to get lock [gateway.go:1211]
Dec 01 13:57:44 sgp1 juicefs[2545041]: Endpoint: http://sgp1.jackal-godzilla.ts.net:9000
Dec 01 13:57:44 sgp1 juicefs[2545041]: Browser Access:
Dec 01 13:57:44 sgp1 juicefs[2545041]:    http://sgp1.jackal-godzilla.ts.net:9000
Dec 01 13:57:44 sgp1 juicefs[2545041]: Object API (Amazon S3 compatible):
Dec 01 13:57:44 sgp1 juicefs[2545041]:    Go:         https://docs.min.io/docs/golang-client-quickstart-guide
Dec 01 13:57:44 sgp1 juicefs[2545041]:    Java:       https://docs.min.io/docs/java-client-quickstart-guide
Dec 01 13:57:44 sgp1 juicefs[2545041]:    Python:     https://docs.min.io/docs/python-client-quickstart-guide
Dec 01 13:57:44 sgp1 juicefs[2545041]:    JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
Dec 01 13:57:44 sgp1 juicefs[2545041]:    .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.609699 juicefs[2545041] <ERROR>: failed to open the file to be locked: .minio.sys/config_lock error permission denied [gateway.go:1300]
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.609918 juicefs[2545041] <ERROR>: failed to open the file to be locked: .minio.sys/config_lock error permission denied [gateway.go:1300]
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.610005 juicefs[2545041] <WARNING>: failed to get lock [gateway.go:1211]
Dec 01 13:57:44 sgp1 juicefs[2545041]: 2024/12/01 06:57:44.610860 juicefs[2545041] <WARNING>: failed to get lock [gateway.go:1211]
Dec 01 13:57:44 sgp1 juicefs[2545041]: IAM initialization complete
Dec 01 14:02:44 sgp1 juicefs[2545041]: 2024/12/01 07:02:44.613336 juicefs[2545041] <WARNING>: failed to get lock [gateway.go:1211]
Dec 01 14:07:44 sgp1 juicefs[2545041]: 2024/12/01 07:07:44.616880 juicefs[2545041] <WARNING>: failed to get lock [gateway.go:1211]

juicefs-gateway.service

# /etc/systemd/system/juicefs-gateway.service
[Unit]
After=redis-juicefs.service tailscaled.service
Description=JuiceFS S3 Gateway

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/kxfhn0s4nzja3mmjbpkn8x1wz6kv2vad-glibc-locales-2.40-36/lib/locale/locale-archive"
Environment="PATH=/nix/store/k48bha2fjqzarg52picsdfwlqx75aqbb-coreutils-9.5/bin:/nix/store/lfhmpajzgfmgl8xrdvp2r9hdi2rf17wy-findutils-4.10.0/bin:/nix/store/14aswfz6pm8zvyyy927xzmi8x4rnvlc8-gnugrep-3.11/bin:/nix/store/gvivnz7rb2gz1vy7d9mdxgzj9didz7w2-gnused-4.9/bin:/nix/store/ivqjhj99firnjq7gp14qf35821viwi5m-systemd-256.7/bin:/nix/store/k48bha2fjqzarg52picsdfwlqx75aqbb-coreutils-9.5/sbin:/nix/store/lfhmpajzgfmgl8xrdvp2r9hdi2rf17wy-findutils-4.10.0/sbin:/nix/store/14aswfz6pm8zvyyy927xzmi8x4rnvlc8-gnugrep-3.11/sbin:/nix/store/gvivnz7rb2gz1vy7d9mdxgzj9didz7w2-gnused-4.9/sbin:/nix/store/ivqjhj99firnjq7gp14qf35821viwi5m-systemd-256.7/sbin"
Environment="TZDIR=/nix/store/218rvnizgdhfkwp9ly4y4ji82lfarcgb-tzdata-2024b/share/zoneinfo"
CacheDirectory=juicefs
CacheDirectoryMode=770
EnvironmentFile=/run/secrets/rendered/juicefsGatewayEnvironmentFile
ExecStart=/nix/store/sacwqx95ra5vbb31kgpv1ik6pmb44kx8-juicefs-1.2.1/bin/juicefs gateway --cache-dir $CACHE_DIRECTORY --cache-mode 0660 redis://sgp1.jackal-godzilla.ts.net:6379 sgp1.jackal-godzilla.ts.net:9000
Group=juicefs
Restart=on-failure
RestartSec=60
RuntimeDirectory=juicefs
RuntimeDirectoryMode=770
StateDirectory=juicefs
StateDirectoryMode=770
User=juicefs
WorkingDirectory=/run/juicefs

[Install]
WantedBy=multi-user.target

WorkingDirectory causes juicefs to start within /run/juicefs (which is RuntimeDirectory)

/etc/passwd

juicefs:x:989:985:system user for the juicefs daemon:/run/juicefs:/run/current-system/sw/bin/nologin

I also set a home directory for user juicefs which is not ideal.

What you expected to happen:

Starts and is able to create its lock file.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version:
  juicefs version 1.2.1+2024-08-30.cd871d19ba57

• Cloud provider or hardware configuration running JuiceFS:

• OS (e.g cat /etc/os-release):
  NixOS 25.05.20241128.970e93b (Warbler) x86_64

• Kernel (e.g. uname -a):
  Linux 6.6.61

• Object storage (cloud provider and region, or self maintained):

• Metadata engine info (version, cloud provider managed or self maintained):

• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
  locally via tailscale

• Others: