Slow download speeds and intermittent crashing due to diskstate reading errors #395
Comments
|
Hi and thank you for the report.
Are you only observing these issues only in version 24.3? Was everything fine with previous versions? |
|
I did play around with the write buffer based on the things I found online during my googling to try and fix the issue on my own. it is set to 1024 based on what I found and the fact that the software itself tells me that is the recommende value for computers with enough memory, I believe it was set to 100 in the stock settings but I could be mistaken there. Should i increase this to 4096? The download rate in settings is set to ''0'', which it tells me is equivelent to unlimited. I could not find the speed control extension in the settings, is this something I would have to add myself? If so, let me know how and where. I observed the issues when downloading real content. Once I observed the issue I started messing around trying to fix it and used the test functionality within the settings to see if it happened there as well or was maybe related to the content I was downloading. So conclusion: yes, the same behaviour happens when downloading actual content. I will try to get a new logfile for you! As for not being able to open it: I had the same issue with standard notepad, however, notepad++ had no such issues and could open the file no problem. Just a tip in the meantime. :) So far it hasnt been crashing the last 24 hours, which might be because I have decreased the amount of queue items? I had it in the multiple thousands (trying to backfill a library), I am adding at most 500 at a time now manually and it seems to atleast not crash now, but obviously this isnt a wanted behaviour. Is it a known issue that NzbGet crashes with large queues in place? The speeds are still low. I will make a new log, but for the crashing it would be useful to get into the already made log I think since its stable in that regard for now. As for the nzbget.conf , I will upload that with the new log. I am not only observing this issue with 24.3. The issue was there with the previous version as well, I was just hoping it might be a known bug that was fixed in 24.3 , hence me trying that when it released yesterday. |
|
Tried increasing the writebuffer to 4096 and did the tests again. Also downloaded the debug version of nzbget. Attached is a ZIP file with new logs, the .conf file (with all usernames passwords removed, I am assuming thats okay?) and a new picture of the settings. Hopefully this, in combination with my previous comment, helps. Speeds are still low, but no crashing. As said in previous comment, I am assuming due to smaller queue based on the behaviour I am seeing, but not certain on that conclusion. Let me know if there is anything else I can do. |
|
But at the same time, speed.cloudflare or fast show the correct speed your ISP provides, right? Thanks for the new logs and for your investigation - I'll do my best to fix these issues asap. |
|
Those two speedtests max out around the 350 Mbps mark. However, when I use a different speedtest (https://www.speedtest.net/), it gives me fullspeed (well, close enough, around the 930-950 Mbps mark). So that may be due to the location of the servers? I use Eweka as my usenet provider, which has its servers in NL where I live, so server location should not be a problem there. But maybe cloudflare and fast dont have servers near me? |
|
Correction: backblaze was running in the background running a backup and hogging my speed lol. Fast and cloudflare now also give me full speed (930-950 ish Mbps). Yes, when testing NzbGet I made sure backblaze was shutdown and not interfering with speeds. |
|
Thank you. According to the screenshot you gave me, the disk speed increased by about 4 times after increasing the Write Buffer, no? Is there any way to confirm the speed issues by testing SABNzb? SABNzb has similar tests whose results could also be useful. Could you share that huge log file again? I'll try to open it on my more powerful laptop. |
|
Huge log file link again: https://file.io/bMlVFIvQKyIk . Like I said, regular notepad just doesnt seem to like big files (tried it on my 5700X desktop, which seems plenty powerful to me). So I used notepad++ which opened it with no problems, might be worth a shot on your end as well! As for trying SABNzb, definitely possible! Do you want me to try a disk speed test? Or should I try with actual content? Both? And when I do, just a test to see what happens or would you like logs and such too? Disk speed increased by about 4 times in the built-in test in settings after increasing the write buffer to 4096, that is correct. However, no difference in actual download performance, still experiencing the slower speeds starting out around the 60 MB/s mark and slowing down quickly to around the 25-30 MB/s mark and eventually to around the 12MB/s mark. |
|
Thank you for another link. Performing SABNzb tests and collecting results is enough. It's also worth trying real content. The Write Buffer option in the disk speed tests is only used during testing, and it's used to find a reasonable value. |
|
Glad you were able to open the big boy log lol. I changed the writebuffer in download queue as well already to 4096, assuming that was needed for the test as well. So that doesnt seem to be the solution for the download speed problem unfortunately.
Did the tests in SABNzb , seems to be working properly there. Also did threw in some content, which is downloading at the 105MB/s mark and dipping to 85MB/s at the lowest and then quickly picks back up to the 105MB/s. Still not full speed, but miles better then whats happening on NZBGet. No idea how SABNzb handles the crashing with big queues, I didnt throw in all my NZB files, just a couple for test purposes. |
|
Thank you. |
|
Thanks! Just FYI, the webinterface is also very slow to load, takes almost half a minute. Not sure if that has anything to do with it, but might be good to know. |
|
Thank you. Please, do not use the Debug version of the app for speed testing and for other performance measures, as it works really slow. The debug version is useful for debugging crashes. Just in case. Researched the logs (the huge ones) and gathered some statistics: As I can see, you updated NZBGet from 21.1 to 24.3 and there were some crashes in both versions: v21.1 crashes: Interrupting? 2 crash: Interrupting? 3 crash Buffer overflow for sure 4 crash Failed article? 5 crash Deleting lots of files? And looks like the update to v24.3 was happening here and after that, the v24.3 crashes: Interrupting? 2 crash Interrupting? 3 crash Interrupting? 4 crash Failed articles? 5 crash Interrupting? 6 crash Faield articles? 7 crash Interrupting?
I'll try to reproduce and fix them. No luck with speed issues so far. I recently fixed an issue with incorrect speed display if the download speed is faster than 2Gb/s and a guy, who raised this topic on Reddit, was happy about it, so it looks really weird to me. |
|
Dont use debug in production, that made sense to me. So I already re-installed the regular version for the speed testing afterwards. If you need debug version logs again please let me know cause I will have to reinstall the debug version again. Yeah, I went from 21.1 to 24.3 for two reasons. One, I found out it was a fork that was still being updated, I had the old NzbGet which i didnt realise , and two, I was hoping it might fix the issues I was having. I did manually delete a bunch of queue files through file explorer, did it either two or three times actually, cause that seemed to fix the whole crashing issue. Possibly corrupt queue files? Just a thought. Although, now that i think about it, if it shows up in the logs it cant be by manual deletion but must be deletion by NzbGet itself? I will try to play around with the number of connections. It seemed logical to use the max allowed by the usenet provider, but maybe it isnt? This is my first time using usenet. I will try your suggestion and report back on that. I do not think its an incorrect speed display in NzbGet in this case personally. Cause if it was, and it was actually going at full speed, then network usage should show full usage. And it isnt. Both task manager and my router config show usage matching what NzbGet is showing me. |
|
|
Reporting back: Played around with the number of connections. Lowering it did lower the transfer speed (maxing out around 10MB/s with 8 connections and around 22MB/s with 16 connections), with 32 connections I was back to my max of around 50MB/s as before and bumping it up from there did nothing. On all amounts of connections it dipped back down again like before, except on 8 connections when it was stable around the 10MB/s mark. So on 8 connections it was stable low, on higher amounts of connections the fluctuation was the same as before. |
|
Yeah, that was predictable. |
|
Just FYI, just loaded up the queue again with almost 1000 items. So we can see whether or not NzbGet really does crash with a large queue, since its been stable (apart from the speeds) the last few days with a smaller queue. I will report back on the results. |
|
Well, its been stable for a few days, even with a big queue...... Why? Who knows..... DId you ever figure out why it crashed constantly before? Would logs for the current good run help? |
|
A few more gigabytes of logs? In general, yes, there can be many reasons for such crashes. |
|
I aim to provide if more nightmares are wanted. :P Sounds like no specific causes were clear yet. Yeah, I will keep an eye on it. :) Might just turn out to be bad / corrupt Nzb files somehow, who knows. |
|
Well, since my last comment its been stable. Havent had any more crashes. Guess this isnt every useful to you as a dev, since we havent found a cause and therefore not a solution, but hey, it works. :) I am just gonna say it got overwhelmed by the queue that was too big, making the queue files corrupt, and then throwing up all over itself. So the ''solution'' seems to be to keep the queue size relatively manageable. Thanks so much for all the help trying to figure this all out! I guess we can close this issue? Unless you want to keep it open and try to figure out the actual cause eventually ofcourse. :) Like I said before, if thats the case and you need any more help / testing, let me know and I will help out where I can. :) |
|
Thank you for keeping an eye on this issue. |
|
You are right, I am still having network speed issues. Although it has been better, its been stable around the 40-50-60 MB/s mark recently. Still not normal, since I have a 1Gbps connection that should be able to sustain 100+, which it still isnt reaching ever, but not dropping down to like 10MB/s anymore. And yeah I agree, its an assumption on my part that that is the issue. But if it is, a heads up ''hey, your queue is way too big, this isnt gonna work'' by the software would help a lot with troubleshooting. Thanks for all the time youre putting into this. :) |
|
v24.3-stable NZBGet will freeze and go to 100% CPU when articles are missing, but binaries that are fully available download normally. Might be related to: #396 |
|
@lars1216-nl |
Just to be certain on what you're asking, you want me to install the latest testing build and then try network speeds right? Should I try with one NZB or try to load it up? If you want me to load it up you're gonna have to wait a few weeks, im out of storage and new HDD's are on the way lol. |
|
@lars1216-nl |
|
I'm actually having the exact same issue. Upgraded to latest version a couple of weeks ago and noticed that my download speed doesn't go over 10mb/s when it normally went to 80mb/s Tested it right now with another nzb downloader using the same amount of connection, same server etc etc. Nzbget getting 7mb/s I have yet to try going back to a previous release or test I did notice that when putting nzbget to using SSL it goes even slower too. Mine is running on docker on an ubuntu VM that's running on proxmox. |
|
What exact version did you update from to the latest? |
|
@dnzbk that is a very good question, I usually don't pull new ones a lot so it has to be 24.0 or earlier. I've just did a few tests by pulling 24.1 but that one is slow too. Haven't checked at even earlier versions. I did just try something that I noticed on the status page. When doing a speed test from the status page I'm getting roughly 400mbps for the 5gb file
The moment I try to download a normal file that's when it starts having an issue. In general this exact setup has been running perfectly fine for the past few years. I did recently switch over to another system for the docker instance. I've only just noticed the speed issue in the past few days, the last time I checked was in the summer and at that point in time speed was fine. I usually don't notice because I only normally let it do its thing during the night and I don't actively monitor the speeds |
|
Thanks. Speed tests show around 50MB/s but when downloading real data the speed varies around 7MB/s? |
Sorry, late response, finally had time to mess about with it for a bit this week. The latest testing build did the trick for me with with both speedtest and a small amount of real files! Once I load it up with a bigger queue it crashes down again, but only to the 40-50MB/s mark. So its majorly improved on the latest testing build! No idea what you did, but it worked on my config. |
|
@lars1216-nl @evilswordfish |
To be clear, it no longer seems to crash with a big queue, just gets quite a bit slower (like i said, slows down to the 40-50MB/s mark, small queue and speedtest do 105-115MB/s which is expected for a 1 gig connection). Could that be expected behaviour maybe? |
I actually tested both before too, both testing branch and less connection. Neither seems to do anything. And yes, speed test shows 50mb/s via the status page with the 5GB package, and then actual files only goes to 6-10mb/s The connections never used to be an issue either since I've always had the 100 configured. I thought maybe it was the system itself having issues but since another NZB downloader works at the expected speed for 60mb/s with the same settings and same files. I currently don't mind the slowness that much seeing that it's configured to work at night but still unexpected behavior. |
Got it, thanks.
No, it is not. |
I see, thanks. |
Is there already an issue for your problem?
NZBGet Version
v24.3-stable
Platform
Windows
Environment
Current Behavior
There are two issues currently.
Transfer speeds are very low. The highest I have seen is around the 60MB/s mark, but it usually hovers around the 12MB/s mark or even lower. Drivepool is properly flushing the NVME drives it uses as cache, so no issues there.
Considering the internet speed (1Gbit) the speed should be higher even when writing to just the mechanical drives (since SATA 3 should do around the 125MB/s mark, even if you half that in real-life use-cases then the 12MB/s speeds still wouldnt make sense).
NzbGet keeps crashing every 12 hours-ish with errrors in the log saying '' error reading diskstate '' followed by specific queue files. When this happens NzbGet completely stops working. Even the WebUI is eternally stuck on ''Loading, please wait''. The only ''solution'' for this is deleting all the queue files for NzbGet and then rebooting NzbGet.
Expected Behavior
Higher speeds, both at first and in sustained workloads.
No complete crashes a few times a day.
Steps To Reproduce
No response
Logs
Cannot include logs since even zipped the 6GB log file is 125MB, which is too big for github unfortunately. Should be able to download them here: https://file.io/gDRoV0kw3ki3 .
Extra information
No response
The text was updated successfully, but these errors were encountered: