404 for every image after upgrade to docker v6

[LouisVallat]

Lychee version

v6.0.0

Which PHP version are you using?

PHP 8.2

Detailed description of the problem

Hi,
After upgrading from a clean v5 instance of Lychee, running on Docker, all the thumbnails and images return a 404, and the whole gallery is broken.

V5 instance had no warning, correct permissions and no issue whatsoever.

v6 says in diagnostics "/uploads/small2x is owned by group www-data, but should be owned by one out of users, lychee" but ls shows :

lychee@836861f59027:/var/www/html/Lychee$ ls -alh /uploads/
total 442K
drwxrwsr-x 13 lychee lychee   13 Jun 13 23:07 .
drwxr-xr-x  1 root   root   4.0K Oct 27 15:17 ..
drwxrwsr-x  2 lychee lychee  819 Oct 27 14:22 big
drwxrwsr-x  2 lychee lychee    3 Mar  6  2024 import
drwxrwsr-x  2 lychee lychee 1.6K Mar  6  2024 medium
drwxrwsr-x  2 lychee lychee    3 Mar  6  2024 medium2x
drwxrwsr-x  2 lychee lychee    3 Mar  6  2024 original
drwxrwsr-x  2 lychee lychee    3 Mar  6  2024 raw
drwxrwsr-x  2 lychee lychee 1.6K Mar  6  2024 small
drwxrwsr-x  2 lychee lychee    3 Mar  6  2024 small2x
drwxrwsr-x  2 lychee lychee 1.6K Mar  6  2024 thumb
drwxrwsr-x  2 lychee lychee    3 Mar  6  2024 thumb2x
drwxrwsr-x  2 lychee lychee    3 Mar  6  2024 tracks
lychee@836861f59027:/var/www/html/Lychee$

I've spent the last 3 hours debugging this issue but no matter what I do everything stays broken 😢

Steps to reproduce the issue

1. Run a working instance of Lychee v5
2. Check diagnostics, no warning whatsoever
3. Update image from lycheeorg/lychee:v5.5.1 to lycheeorg/lychee:v6.0.0 in docker-compose.yml
4. docker compose up -d
5. No images are displayed

Newly uploaded images are displayed correctly, but previously uploaded ones are not.

Diagnostics [REQUIRED]

Diagnostics
-----------
Warning: /v**********************lic/uploads/ is owned by group www-data, but should be owned by one out of users, lychee
Warning: /v**********************lic/uploads/ has permissions 0755, but should have 2775
Error: /v**********************lic/uploads/ is not writable by users, lychee
Warning: /v**********************lic/uploads/thumb is owned by group www-data, but should be owned by one out of users, lychee
Warning: /v**********************lic/uploads/thumb has permissions 0755, but should have 2775
Error: /v**********************lic/uploads/thumb is not writable by users, lychee
Warning: /v**********************lic/uploads/original is owned by group www-data, but should be owned by one out of users, lychee
Warning: /v**********************lic/uploads/original has permissions 0755, but should have 2775
Error: /v**********************lic/uploads/original is not writable by users, lychee
Warning: /v**********************lic/uploads/medium2x is owned by group www-data, but should be owned by one out of users, lychee
Warning: /v**********************lic/uploads/medium2x has permissions 0755, but should have 2775
Error: /v**********************lic/uploads/medium2x is not writable by users, lychee
Warning: /v**********************lic/uploads/small2x is owned by group www-data, but should be owned by one out of users, lychee
Warning: /v**********************lic/uploads/small2x has permissions 0755, but should have 2775
Error: /v**********************lic/uploads/small2x is not writable by users, lychee
Warning: 5 more directories with wrong owner
Warning: 5 more directories with wrong permissions
Warning: 5 more inaccessible directories
Warning: You may experience problems when uploading a photo of large size. Take a look in the FAQ for details.
Warning: You may experience problems when uploading a photo of large size. Take a look in the FAQ for details.
Info: Latest version of PHP is 8.3
Info: git (software) is not available.
Info: Have you considered supporting Lychee? :)

System Information
------------------
Lychee Version (tag):                      v6.0.0 (d25cd64) -- Up to date (-2.3496148528666E-6 years ago).
DB Version:                                6.0.0

composer install:                          --no-dev
APP_ENV:                                   production
APP_DEBUG:                                 false
APP_URL:                                   set
APP_DIR:                                   default
LOG_VIEWER_ENABLED:                        true
VUEJS_ENABLED:                             true
PHOTO_PIPES:                               true

System:                                    Linux
PHP Version:                               8.2.24
PHP User agent:                            Lychee/4 (https://lycheeorg.github.io/)
Timezone:                                  Europe/Paris
Max uploaded file size:                    2M
Max post size:                             8M
Chunk size:                                0.00 B
Max execution time:                        0
PostgreSQL Version:                        PostgreSQL 17.0 (Debian 17.0-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit

exec() Available:                          yes
Imagick Available:                         1
Imagick Enabled:                           1
Imagick Version:                           1691
GD Version:                                2.3.3
Number of foreign key:                     12 found.

Config Information
------------------
version:                                   060000
check_for_updates:                         0
sorting_photos_col:                        taken_at
sorting_photos_order:                      DESC
sorting_albums_col:                        max_taken_at
sorting_albums_order:                      DESC
imagick:                                   1
skip_duplicates:                           0
small_max_width:                           0
small_max_height:                          360
medium_max_width:                          1920
medium_max_height:                         1080
lang:                                      en
image_overlay_type:                        exif
default_license:                           reserved
compression_quality:                       90
grants_full_photo_access:                  1
delete_imported:                           0
mod_frame_enabled:                         1
mod_frame_refresh:                         30
thumb_2x:                                  1
small_2x:                                  1
medium_2x:                                 1
landing_page_enable:                       0
force_migration_in_production:             0
search_public:                             1
hide_version_number:                       1
SL_enable:                                 0
SL_for_admin:                              0
recent_age:                                1
grants_download:                           0
photos_wraparound:                         1
map_display:                               1
zip64:                                     1
map_display_public:                        1
map_provider:                              Wikimedia
force_32bit_ids:                           0
map_include_subalbums:                     0
update_check_every_days:                   3
has_exiftool:                              1
share_button_visible:                      0
import_via_symlink:                        0
has_ffmpeg:                                1
location_decoding:                         0
location_decoding_timeout:                 30
location_show:                             1
location_show_public:                      0
rss_enable:                                0
rss_recent_days:                           7
rss_max_items:                             100
prefer_available_xmp_metadata:             0
editor_enabled:                            1
lossless_optimization:                     0
swipe_tolerance_x:                         150
swipe_tolerance_y:                         250
local_takestamp_video_formats:             .avi|.mov
log_max_num_line:                          1000
unlock_password_photos_with_url_param:     0
nsfw_visible:                              1
nsfw_blur:                                 0
nsfw_warning:                              0
nsfw_warning_admin:                        0
nsfw_banner_override:
map_display_direction:                     1
album_subtitle_type:                       oldstyle
upload_processing_limit:                   4
new_photos_notification:                   0
legacy_id_redirection:                     1
zip_deflate_level:                         6
default_album_protection:                  1
allow_username_change:                     1
album_decoration:                          layers
album_decoration_orientation:              row
auto_fix_orientation:                      1
use_job_queues:                            0
random_album_id:                           starred
use_last_modified_date_when_no_exif_date:  0
layout:                                    justified
date_format_photo_thumb:                   M j, Y, g:i:s A e
date_format_photo_overlay:                 M j, Y, g:i:s A e
date_format_sidebar_uploaded:              M j, Y, g:i:s A e
date_format_sidebar_taken_at:              M j, Y, g:i:s A e
date_format_hero_min_max:                  F Y
date_format_hero_created_at:               M j, Y, g:i:s A T
date_format_album_thumb:                   M Y
upload_chunk_size:                         0
nsfw_banner_blur_backdrop:                 0
search_pagination_limit:                   1000
search_minimum_length_required:            4
photo_layout_justified_row_height:         320
photo_layout_masonry_column_width:         300
photo_layout_grid_column_width:            250
photo_layout_square_column_width:          200
photo_layout_gap:                          12
display_thumb_album_overlay:               always
display_thumb_photo_overlay:               hover
default_album_thumb_aspect_ratio:          1/1
use_album_compact_header:                  0
login_button_position:                     left
back_button_enabled:                       0
back_button_text:                          Return to Home
current_job_processing_visible:            1
SA_random_thumbs:                          0
job_processing_queue_visible:              1
login_required:                            0
enable_unsorted:                           1
enable_starred:                            1
enable_recent:                             1
enable_on_this_day:                        1
login_required_root_only:                  1
dark_mode_enabled:                         1
show_keybinding_help_popup:                0
show_keybinding_help_button:               1
disable_se_call_for_actions:               0
enable_se_preview:                         0
slideshow_timeout:                         5
keep_original_untouched:                   1
hide_nsfw_in_smart_albums_and_search:      1

Browser & System [REQUIRED]

Up-to-date Fedora 40 and Firefox.

Application running on Docker inside an up-to-date Debian 12 VM.

Please confirm (incomplete submissions will not be addressed)

• I have provided easy and step-by-step instructions to reproduce the bug.
• I understand my bug report will be removed if I haven't met the criteria above.

[ildyria]

I smell something related to user permissions on your docker install / docker compose?
I don't think this is specific to the migration, because I literally did the same on my install and it didn't crash.

[ildyria]

If I log into my container I get:

▶ docker exec -it 9b2 sh                     
# ls
Banner.png	 README.md    bootstrap       composer.lock	  docker_target      package.json    phpunit.xml	routes			    storage		vendor
CONTRIBUTING.md  SECURITY.md  codecov.yml     config		  index.php	     phpstan	     postcss.config.js	scripts			    tailwind.config.ts	version.md
LICENSE		 app	      composer-cache  database		  lang		     phpstan.neon    public		simple_error_template.html  tests		vite.config.ts
Makefile	 artisan      composer.json   developer-notes.md  package-lock.json  phpunit.ci.xml  resources		sonar-project.properties    tsconfig.json
# cd public
# ls -lola
total 100
drwxr-xr-x 1 www-data  4096 Oct 27 14:50 .
drwxr-xr-x 1 www-data  4096 Oct 27 14:50 ..
-rw-r--r-- 1 www-data  1908 Oct 26 21:04 .htaccess
-rw-r--r-- 1 www-data   387 Oct 26 21:04 .user.ini
drwxr-xr-x 2 www-data  4096 Oct 26 21:04 Lychee-front
drwxr-xr-x 3 www-data  4096 Oct 26 21:06 build
drwxr-xr-x 1 www-data  4096 Oct 27 14:50 dist
-rw-r--r-- 1 www-data 33412 Oct 26 21:04 favicon.ico
drwxr-xr-x 2 www-data  4096 Oct 26 21:04 fonts
drwxr-xr-x 2 www-data  4096 Oct 26 21:04 img
-rw-r--r-- 1 www-data  2614 Oct 26 21:04 index.php
drwxr-xr-x 3 www-data  4096 Oct 26 21:04 installer
-rw-r--r-- 1 www-data   254 Oct 26 21:04 robots.txt
lrwxrwxrwx 1 www-data    21 Oct 26 21:04 storage -> ../storage/app/public
lrwxrwxrwx 1 root         4 Oct 27 14:50 sym -> /sym
lrwxrwxrwx 1 root         8 Oct 27 14:50 uploads -> /uploads
drwxr-xr-x 3 www-data  4096 Oct 26 21:04 vendor
-rw-r--r-- 1 www-data  1193 Oct 26 21:04 web.config
# cd uploads    
# ls -lola
total 18293200
drwxrwsr-x  14 lychee        4096 Sep 18 06:41 .
drwxr-xr-x   1 root          4096 Oct 27 14:49 ..
drwxrwsr-x   2 lychee     2154496 Aug 26 11:26 big
-rw-rw-r--   1 lychee    58559212 Sep 18 09:08 dump.sql
drwxrwsr-x   2 lychee        4096 Dec  3  2023 import
drwxrwsr-x   2 lychee        4096 Sep 26 08:42 livewire-tmp
drwxrwsr-x 258 lychee     2088960 Dec  3  2023 medium
drwxrwsr-x   2 lychee        4096 Jan 11  2023 medium2x
drwxrwsr-x 258 lychee        4096 Jan 11  2023 original
drwxrwsr-x   2 lychee        4096 Jan 11  2023 raw
drwxrwsr-x 258 lychee     2187264 Aug 26 11:26 small
drwxrwsr-x   2 lychee        4096 Jan 11  2023 small2x
drwxrwsr-x 258 lychee     4718592 Aug 26 11:26 thumb
drwxrwsr-x 258 lychee        4096 Jan 11  2023 thumb2x
drwxrwsr-x   2 lychee        4096 Jan 11  2023 tracks
-rw-rw-r--   1 lychee 18662461440 Sep 18 06:34 uploads.tar

Then checking the groups:

# groups lychee
lychee : lychee users
# groups www-data
www-data : www-data lychee

[LouisVallat]

Thank you for your quick reply!
I apologize if my non-native English gave you the impression that I was saying the migration was the issue 😅 It may well not be, just that there were no prior warning on 5.5.1 and just after updating everything was broken.

I redid your steps, and that's what I obtained:

root@lychee:/srv/gallery_louis_vallat_dev# docker compose exec app sh
# ls
Banner.png	 composer-cache      package.json	simple_error_template.html
CONTRIBUTING.md  composer.json	     phpstan		sonar-project.properties
LICENSE		 composer.lock	     phpstan.neon	storage
Makefile	 config		     phpunit.ci.xml	tailwind.config.ts
README.md	 database	     phpunit.xml	tests
SECURITY.md	 developer-notes.md  postcss.config.js	tsconfig.json
app		 docker_target	     public		vendor
artisan		 index.php	     resources		version.md
bootstrap	 lang		     routes		vite.config.ts
codecov.yml	 package-lock.json   scripts
# cd public
# ls -lola
total 104
drwxr-xr-x  1 www-data  4096 Oct 27 17:23 .
drwxr-xr-x  1 www-data  4096 Oct 27 17:23 ..
-rw-r--r--  1 www-data  1908 Oct 26 23:04 .htaccess
-rw-r--r--  1 www-data   387 Oct 26 23:04 .user.ini
drwxr-xr-x  2 www-data  4096 Oct 26 23:04 Lychee-front
drwxr-xr-x  3 www-data  4096 Oct 26 23:06 build
drwxr-xr-x  1 www-data  4096 Oct 27 17:23 dist
-rw-r--r--  1 www-data 33412 Oct 26 23:04 favicon.ico
drwxr-xr-x  2 www-data  4096 Oct 26 23:04 fonts
drwxr-xr-x  2 www-data  4096 Oct 26 23:04 img
-rw-r--r--  1 www-data  2614 Oct 26 23:04 index.php
drwxr-xr-x  3 www-data  4096 Oct 26 23:04 installer
-rw-r--r--  1 www-data   254 Oct 26 23:04 robots.txt
lrwxrwxrwx  1 www-data    21 Oct 26 23:04 storage -> ../storage/app/public
lrwxrwxrwx  1 root         4 Oct 27 17:23 sym -> /sym
drwxr-xr-x 11 www-data  4096 Oct 26 23:04 uploads
drwxr-xr-x  3 www-data  4096 Oct 26 23:04 vendor
-rw-r--r--  1 www-data  1193 Oct 26 23:04 web.config
# cd uploads
# ls -lola
total 48
drwxr-xr-x 11 www-data 4096 Oct 26 23:04 .
drwxr-xr-x  1 www-data 4096 Oct 27 17:23 ..
drwxr-xr-x  2 www-data 4096 Oct 26 23:04 import
drwxr-xr-x  2 www-data 4096 Oct 26 23:04 medium
drwxr-xr-x  2 www-data 4096 Oct 26 23:04 medium2x
drwxr-xr-x  2 www-data 4096 Oct 26 23:04 original
drwxr-xr-x  2 www-data 4096 Oct 26 23:04 small
drwxr-xr-x  2 www-data 4096 Oct 26 23:04 small2x
drwxr-xr-x  2 www-data 4096 Oct 26 23:04 thumb
drwxr-xr-x  2 www-data 4096 Oct 26 23:04 thumb2x
drwxr-xr-x  2 www-data 4096 Oct 26 23:04 tracks

As for the permissions:

# groups lychee
lychee : lychee users
# groups www-data
www-data : www-data lychee

Looks like the issue might be that uploads isn't symlink-ed to /uploads, causing the 404 and the permission issues 🤔

The /uploads folder is mounted correctly though:

# ls -lola /uploads
total 442
drwxrwsr-x 13 lychee   13 Jun 13 23:07 .
drwxr-xr-x  1 root   4096 Oct 27 17:23 ..
drwxrwsr-x  2 lychee  819 Oct 27 14:22 big
drwxrwsr-x  2 lychee    3 Mar  6  2024 import
drwxrwsr-x  2 lychee 1590 Mar  6  2024 medium
drwxrwsr-x  2 lychee    3 Mar  6  2024 medium2x
drwxrwsr-x  2 lychee    3 Mar  6  2024 original
drwxrwsr-x  2 lychee    3 Mar  6  2024 raw
drwxrwsr-x  2 lychee 1634 Mar  6  2024 small
drwxrwsr-x  2 lychee    3 Mar  6  2024 small2x
drwxrwsr-x  2 lychee 1635 Mar  6  2024 thumb
drwxrwsr-x  2 lychee    3 Mar  6  2024 thumb2x
drwxrwsr-x  2 lychee    3 Mar  6  2024 tracks

The startup logs during the migration are those:

Attaching to app-1
app-1  |
app-1  | -------------------------------------
app-1  |   _               _
app-1  |  | |   _   _  ___| |__   ___  ___
app-1  |  | |  | | | |/ __|  _ \ / _ \/ _ \
app-1  |  | |__| |_| | (__| | | |  __/  __/
app-1  |  |_____\__, |\___|_| |_|\___|\___|
app-1  |        |___/
app-1  |
app-1  | -------------------------------------
app-1  | Lychee Version: 6.0.0 (release)
app-1  | Lychee Branch:  master
app-1  | Lychee Commit:  d25cd64
app-1  | https://github.com/LycheeOrg/Lychee/commit/d25cd64e08680d30c62bab8dc2a837ef2ecabc7b
app-1  | -------------------------------------
app-1  | **** Make sure the /conf /uploads /sym /logs /lychee-tmp folders exist ****
app-1  | **** Create the symbolic link for the /uploads folder ****
app-1  | cp: cannot create regular file '/uploads/import/index.html': Permission denied
app-1  | cp: cannot create regular file '/uploads/medium/index.html': Permission denied
app-1  | cp: cannot create regular file '/uploads/medium2x/index.html': Permission denied
app-1  | cp: cannot create regular file '/uploads/original/index.html': Permission denied
app-1  | cp: cannot create regular file '/uploads/small/index.html': Permission denied
app-1  | cp: cannot create regular file '/uploads/small2x/index.html': Permission denied
app-1  | cp: cannot create regular file '/uploads/thumb/index.html': Permission denied
app-1  | cp: cannot create regular file '/uploads/thumb2x/index.html': Permission denied
app-1  | cp: cannot create regular file '/uploads/tracks/index.html': Permission denied
app-1  | **** Create the symbolic link for the /sym folder ****
app-1  | **** Create the symbolic link for the /logs folder ****
app-1  | **** Create the symbolic link for the /lychee-tmp folder ****
app-1  | **** Copy the .env to /conf ****
app-1  | **** Inject .env values ****
app-1  | **** Generate the key (to make sure that cookies cannot be decrypted etc) ****
app-1  |
app-1  |
app-1  |                            APPLICATION IN PRODUCTION.
app-1  |
app-1  |
app-1  |    WARN  Command cancelled.
app-1  |
app-1  | **** Migrate the database ****
app-1  |
app-1  |    INFO  Running migrations.
app-1  |
app-1  |   2024_07_06_214241_thumb_min_max_take_date_order ................ 3.38ms DONE
app-1  |   2024_07_12_183751_add_auto_play_config ......................... 0.81ms DONE
app-1  |   2024_07_26_120007_simplify_config .............................. 0.69ms DONE
app-1  |   2024_07_29_172018_fix_settings ................................. 3.52ms DONE
app-1  |   2024_08_09_205532_white_theme .................................. 0.76ms DONE
app-1  |   2024_08_31_090626_config_help_popup ............................ 0.80ms DONE
app-1  |   2024_08_31_090740_fix_description_always_hover_hidden_photo .... 1.16ms DONE
app-1  |   2024_09_14_143949_add_settings_level ........................... 2.73ms DONE
app-1  |   2024_09_14_153948_fix_more_descriptions ........................ 5.01ms DONE
app-1  |   2024_09_27_144741_add_supporter_fields ......................... 0.81ms DONE
app-1  |   2024_10_05_125328_add_slideshow_timeout ........................ 0.71ms DONE
app-1  |   2024_10_05_125833_lang_is_admin_setting ........................ 0.75ms DONE
app-1  |   2024_10_05_184315_bump_version060000 ........................... 0.75ms DONE
app-1  |   2024_10_09_191528_image_processing_backup ...................... 0.74ms DONE
app-1  |   2024_10_09_194010_image_processing_details_and_others .......... 1.33ms DONE
app-1  |   2024_10_10_101333_set_dropbox_disabled ......................... 0.78ms DONE
app-1  |   2024_10_11_173054_add_space_quota_per_user ..................... 1.17ms DONE
app-1  |   2024_10_11_173106_add_space_quota_configuration ................ 0.75ms DONE
app-1  |   2024_10_13_134419_oath_group_to_users .......................... 0.95ms DONE
app-1  |   2024_10_14_104644_show_nsfw_in_smart_albums .................... 0.78ms DONE
app-1  |   2024_10_14_104823_refine_text_enable_unsorted_smart_album ...... 0.83ms DONE
app-1  |   2024_10_20_125449_move-socials-to-footer ....................... 0.83ms DONE
app-1  |   2024_10_20_173904_add_photo_layout_per_album ................... 0.86ms DONE
app-1  |   2024_10_23_222857_change_header ................................ 0.40ms DONE
app-1  |   2024_10_23_225332_warning_html_content ......................... 4.86ms DONE
app-1  |
app-1  | **** Make sure user.css exists and symlink it ****
app-1  | **** Make sure custom.js exists and symlink it ****
app-1  | **** Create user and use PUID/PGID ****
app-1  |  	User UID :	3002
app-1  |  	User GID :	3002
app-1  | **** Make sure Laravel's log exists ****
app-1  | **** Set Permissions ****
app-1  | **** Start cron daemon ****
app-1  | Starting periodic command scheduler: cron.
app-1  | **** Setup complete, starting the server. ****
app-1  | 192.168.254.2 - - [27/Oct/2024:17:23:22 +0100] "GET / HTTP/1.0" 200 71730 "-" "Uptime-Kuma/1.23.15"

I will try to debug the symlink issue on my side, will come back as soon as I have more clues 😄

[LouisVallat]

OK, I think I have identified the root cause for this permission issue.

• My uploads folder is an NFS mount, with only lychee being able to do anything on the share.
• The entrypoint attempts to write as root in the said share when doing its cp -r /var/www/html/Lychee/public/uploads/* /uploads step
• The step is failing, thus never doing the next part, being making the symbolic link in itself.
• The symbolic link not being made, no image could be accessed.

I confirmed this theory by doing each steps manually, and once the ln operation being made manually, all images start loading as they should.

TLDR: the issue is that the container, upon starting, runs some operations as root before dropping its privileges to the user specified, which is not allowed on my setup (NFS share with strict permissions).

I'll try to allow root operations on this share.

[LouisVallat]

Well, looks like it's fixed 🥳 the issue was that root wasn't allowed to write in the NFS share.

For anyone coming across this issue in the future:
By default, NFS share "squash" root privileges, and root gets mapped as a guest user, from the share's point of view.

You can use an option called no_root_squash on bare NFS setups, which corresponds to a maproot user and group permissions in TrueNAS such as:

This was the fix for me, no errors anymore in boot script, and no issue found in diagnostics!
Thank you again, I really love Lychee, and I've been using it for quite a few years, keep up the good work 💯