Performance improvement mount www

[luukverhoeven]

Moodle has many files, when docker mounts your local files system it has bad performance by default (tested on Intel mac and M1).

Did someone test to implement https://github.com/EugenMayer/docker-sync into this project or another way to improve the speed?

--
Today I did a test with docker-sync the performance were much better.

[scara]

Hi @luukverhoeven,
could you give #183 a chance?

HTH,
Matteo

[luukverhoeven]

Thanks for the tip @scara! It doesn't work directly out of the box.
I needed to start the NFS service, allow access to directories and allow docker to mount it.

After that, I remove the prefix.

After this, it works.
Only performance isn't great in comparison to docker-sync. It's a faster solution then default.

This benchmarks of a Drupal test in docker https://www.jeffgeerling.com/blog/2020/revisiting-docker-macs-performance-nfs-volumes looks also interesting. docker-bg-sync is also a good alternative mount driver https://github.com/cweagans/docker-bg-sync

For testing, docker-bg-sync I used the following setup. Doesn't need any extra permissions or settings.

volumes-nfs.yml

# Here we support https://vivait.co.uk/labs/docker-for-mac-performance-using-nfs
# for improved performance on mac
version: "2"
services:
  webserver:
    volumes:
        - /var/www/html

  bg-sync:
    image: cweagans/bg-sync
    volumes:
      - "${MOODLE_DOCKER_WWWROOT}:/source"
    volumes_from:
      - webserver
    environment:
      - SYNC_DESTINATION=/var/www/html
      - SYNC_MAX_INOTIFY_WATCHES=40000
      - SYNC_VERBOSE=1
    privileged: true

Benchmark on macOS Big Sure M1 (docker 20.10.8)

docker-bg-sync

The /my page:
Page loads in 0.129706 secs

Creating an M course:

19.3 secs

NFS

The /my page:
Page loads between 0.935715 secs and 3.088661 secs. This can be almost 24 times slower.

Creating an M course:

19.6 secs

The most significant difference is when requesting pages with NFS or docker-bg-sync.
DB read and writes are the same, they are already stored on a docker container.

[scara]

Hi @odeialba,
since you are the author of the proposal in #183, what do you think about the @luukverhoeven's proposal?

Disclaimer: I'm a Linux/Windows user, no mac on my side.

TIA,
Matteo

[odeialba]

Hi @luukverhoeven,
I tried using docker-bg-sync and I couldn't make it work.
The new bg-sync container contains the correct files in /source directory, but then /var/www/html only contains one package-lock.json file.
The webserver container also contains one package-lock.json file in /var/www/html directory.

These are the logs in bg-sync container:

==> Starting bg-sync

==> Configuration:
-----> SYNC_SOURCE:                  /source
-----> SYNC_DESTINATION:             /var/www/html
-----> SYNC_VERBOSE:                 1
-----> SYNC_MAX_INOTIFY_WATCHES:     40000

==> Attempting to set maximum inotify watches to 40000
-----> If the container exits with 'Operation not allowed', make sure that
-----> the container is running in privileged mode.
fs.inotify.max_user_watches=40000
fs.inotify.max_user_watches = 40000

==> Generating Unison profile

==> Starting continuous sync.
Warning: No archive files were found for these roots, whose canonical names are:
/source
/var/www/html
This can happen either
because this is the first time you have synchronized these roots, 
or because you have upgraded Unison to a new version with a different
archive format.  

It is important to point out that if I change - "${MOODLE_DOCKER_WWWROOT}:/source" for - ".:/source" it works (but of course, that's not the directory I want).

[luukverhoeven]

@odeialba Did you give the docker-bg-sync container sometime to sync? All files will be synced to this container and this takes a moment.

Also, your docker need access to the Moodle directory. When you use docker desktop, you can find it here.

[odeialba]

@luukverhoeven I still couldn't make it work.
This is the repository and branch I'm using https://github.com/odeialba/moodle-docker/tree/wp
To test it, just comment out the line 152 in bin/moodle-docker-compose and uncomment the line 153
Let me know if it works for you. The error might be caused for some changes I did there

[luukverhoeven]

@odeialba

Strange, I tested your branch, and it also works here.

export MOODLE_DOCKER_WWWROOT=/Users/luukverhoeven/OPENSOURCE/moodle

cd /Users/luukverhoeven/OPENSOURCE/moodle_docker_clone
bin/moodle-docker-compose up -d

[odeialba]

@luukverhoeven I understand that that is the bg-sync container.
Could you please ls /var/www/html directory?
And then ls /var/www/html directory in the webserver container?

[luukverhoeven]

That works, but it takes time (+/- 1 minute):

[odeialba]

@luukverhoeven No matter how long I wait, it does not change:
bg-sync container:

webserver container:

Result:

[luukverhoeven]

@odeialba Which directory did you mount on MOODLE_DOCKER_WWWROOT? I see your /source folder contains adminer , lms, wp ?

[luukverhoeven]

Can you check if the following config helps?

volumes-sync.yml

version: "2"
services:
  webserver:
    volumes:
      - /var/www/html

  bg-sync:
    image: kitertjuh/docker-bg-sync
    volumes:
      - "${MOODLE_DOCKER_WWWROOT}:/source"
    volumes_from:
      - webserver
    environment:
      - SYNC_DESTINATION=/var/www/html
      - SYNC_MAX_INOTIFY_WATCHES=40000
      - SYNC_VERBOSE=1
    privileged: true

---

[odeialba]

@odeialba Which directory did you mount on MOODLE_DOCKER_WWWROOT? I see your /source folder contains adminer , lms, wp ?

@luukverhoeven That directory contains two instances of moodle (wp and lms) and one of adminer.
If everything worked, in the browser I should see the three directories.

I tried your suggestion, but the result is exactly the same.

[luukverhoeven]

Maybe it's caused by the 3 directory structure and many files.
Did you try only mounting Moodle?

[odeialba]

@luukverhoeven Still not working. The only difference is that /source directory in bg-sync container contains all the moodle files, but they are not synced to the /var/www/html directory

[luukverhoeven]

Can you give me more information about your system, docker version / config , directory location etc.
Then I can try to replicate your issue.

[odeialba]

@luukverhoeven

• System:
  macOS Big Sur
  version 11.6
  Intel i5 processor
• Docker desktop:
  Version 4.0.1 (68347)
  Engine 20.10.8
• Docker Resources:
  CPUs: 4
  Memory: 4.00GB
  Swap: 2GB
  Disk image size: 59.6 GB (21.9 GB used)
  Disk image location: /Users/odeialba/Library/Containers/com.docker.docker/Data/vms/0/data
• Docker network: 192.168.65.0/24
• Docker Engine configuration:

{
  "experimental": false,
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "features": {
    "buildkit": true
  }
}

• Variables:
  MOODLE_DOCKER_WWWROOT=/Users/odeialba/Projects/Moodle/moodle
  MOODLE_DOCKER_DB=pgsql
  MOODLE_DOCKER_DBTYPE=pgsql
  MOODLE_DOCKER_WEB_HOST=moodle.local
  MOODLE_DOCKER_PHP_VERSION=7.4
  MOODLE_DOCKER_WEB_PORT=80
  MOODLE_DOCKER_SELENIUM_VNC_PORT=54321

[luukverhoeven]

@odeialba Thanks for the information. I will investigate this soon on a new VM with macOS Big Sur.

[luukverhoeven]

@odeialba I recorded the debugging on the VM.
You can see it here: https://www.youtube.com/watch?v=GTioX1-3k0Y

The VM is a fresh installation of macOS Big Sur and a default installation of docker desktop.
I used your branch and settings.

The conclusion is that patience is the key ;)

[odeialba]

@luukverhoeven I finally made it work (FYI I couldn't make grunt watch work in any of the versions) but I didn't feel any big difference plus grunt is not working with bg-sync at all. What tests are you running? Maybe I could run them as well to see the difference.

[natewallis]

@odeialba The speed of this moodle docker instance inside moodle is very slow. The mailhog UI responds immediately, so I assume it is just due to the number of files in the moodle source folder? I have no experience with docker-sync, but I am also trying to have as simple configuration as possible so I can expose my work peers to Docker.... any recommendations how to deal with thi?

[odeialba]

@nwallis Try with these changes https://github.com/moodlehq/moodle-docker/pull/183/files

[mplt-aleks]

@nwallis Try with these changes https://github.com/moodlehq/moodle-docker/pull/183/files

Hey yah @odeialba! Thanks a lot for this. Btw, do I need to run the nfs script every time I up moodle docker? Thanks again mate! :D

[odeialba]

Hi @mplt-aleks
It only needs to be run once

[mplt-aleks]

Hey @odeialba. Alright got that. I'm currently trying this now and man the moodle installation took just 1 or 2 minutes unlike before it was about 20 minutes to finished. I'm running M1 Pro chip and I've been struggling with the smooth setup. Cheers!

[mouneyrac]

The current Mac performance solution is to enable both the experimental settings (virtualisation and VirtioFS) in the preferences of Docker Desktop.

See https://www.docker.com/blog/speed-boost-achievement-unlocked-on-docker-desktop-4-6-for-mac/

(it will boost your performance like crazy - no more 40mn docker install)

[mplt-aleks]

Hi @odeialba, the NFS solution doesn't seem to work anymore with these recent macOS and Docker updates. Do you still have yours working?

Tried a fresh install, still no luck.

Was planning to do the same above(@mouneyrac ), the enabling of VirtioFS, but docker won't start when it's on(true). Had to modify Docker's settings.json to turn it off(false) and make docker start again.

MacOS: 12.6
Docker: 4.12.0

⠹ Container moodle-nfs-webserver-1 Creating 0.1s Error response from daemon: failed to mount local volume: mount :/System/Volumes/Data/./moodle_src/:/var/lib/docker/volumes/moodle-nfs_nfsmount/_data, data: addr=192.168.65.2,nolock,hard,nointr,nfsvers=3: no such file or directory