Docker crashes about every 8 days - thousands of no such container

[bignay2000]

⚠️ Please verify that this bug has NOT been reported before.

• I checked and didn't find similar issue

🛡️ Security Policy

• I agree to have read this project Security Policy

Description

Docker daemon restarts about every 8 days. This has occurred 3 times and started when I first installed dockge.

Currently running v. 1.4.1

I think Dockge is trying to collect stats thousands of times for a container that no longer exists.

Jan 16 07:12:09 env[1302]: time="2024-01-16T07:12:09.927616462Z" level=error msg="collecting stats for d5721589ab83c509fb8f0ada3ef94034e7d07717640a01cfaa4228567c0a37a8: no such container"

journalctl -xu docker.service | grep "d5721589ab83c509fb8f0ada3ef94034e7d07717640a01cfaa4228567c0a37a8: no such container" | wc -l
263204

👟 Reproduction steps

Run dockge with unmanaged containers

👀 Expected behavior

dockge should rate limit itself if container not found or stop trying to get metrics for containers that do not exist.

😓 Actual Behavior

Apparently Dockge is in a loop trying to collect metrics for a container that no longer exists.

Dockge Version

1.4.1

💻 Operating System and Arch

Flatcar Container Linux by Kinvolk 3760.2.0 (Oklo) X64

🌐 Browser

Google Chrome

🐋 Docker Version

Docker version 20.10.24, build e78084afe5

🟩 NodeJS Version

No response

📝 Relevant log output

Jan 27 05:05:08 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:05:08.432059156Z" level=error msg="col>
Jan 27 05:05:09 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:05:09.458339342Z" level=error msg="col>
Jan 27 05:05:10 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:05:10.513376089Z" level=error msg="col>
Jan 27 05:05:11 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:05:11.552601973Z" level=error msg="col>
Jan 27 05:05:12 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:05:12.584186249Z" level=error msg="col>
Jan 27 05:05:13 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:05:13.702135573Z" level=error msg="col>
Jan 27 05:05:14 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:05:14.750881971Z" level=error msg="col>
Jan 27 05:05:15 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:05:15.793704552Z" level=error msg="col>
Jan 27 05:05:16 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:05:16.876216988Z" level=error msg="col>
Jan 27 05:16:29 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:16:29.691554245Z" level=warning msg="E>
Jan 27 05:16:29 hivevm.hivetechnologies.net env[1351]: time="2024-01-27T05:16:29.691581095Z" level=info msg="Atte>

[bignay2000]

Seems similar to moby/moby#31655

[bignay2000]

Updated from 1.4.1 to the latest 1.4.2 after the last crash @ 2024-01-27 00:04:26.
1.4.2 may have fixed this occurrence.

#380 - deleted tmp folder on startup

So far no errors reported in the last hour.

journalctl -xu docker.service --since "1 hour ago"

[bignay2000]

Created a new monitor to check every hour for docker service errors on all my docker hosts. I report back if this reoccurs with the latest 1.4.2 release.

#!/bin/bash
set -E -u -o pipefail -e

sudo journalctl -xu docker.service --since "1 hour ago"
errorcount=$(sudo journalctl -xu docker.service --since "1 hour ago" | awk /level=error/ | wc -l)

if [[ $errorcount -gt 0 ]]
then
  echo "Found errors in Docker Service"
  exit 1
else
  echo "No errors found in Docker Service in the last hour"
fi

[bignay2000]

Has not reproduced after 1.4.2 update.

[louislam]

Should be due to the #380 issue, feel free to report again if you see this issue again.

[dacardona]

Hello everyone.

I'm not using dockge, but just own services, every 5 or 8 days all services (differents apps from own apps to dbs, queues, etc) for all nodes are re-created, when I inspect the services (docker service ps) the error is "No such container" but why?

I'm running Docker 26.0.0 on Ubuntu 20.04.6 in a Swarm cluster of 3 nodes