ZHA - application controller is not running #130548
Comments
|
Hey there @dmulcahey, @Adminiuga, @puddly, @TheJulianJES, mind taking a look at this issue as it has been labeled with an integration ( Code owner commandsCode owners of
(message by CodeOwnersMention) zha documentation |
erkr
changed the title
|
My ZHA network has gone down twice now with similar issues, also starting with core 2024.11.1. |
|
@Nerwyn also using a zbdongle-e?! |
|
Nope, using the SONOFF Zigbee 3.0 USB Dongle Plus-E Gateway with Silicon Labs Multiprotocol. |
|
@Nerwyn Multiprotocol isn't really a supported configuration anymore. I strongly suggest you migrate back to Zigbee firmware and explore alternative options for Thread border routers (if you use Thread). |
|
@puddly For some reason I see ZHA resetting the dongle as it is reported in fail state. That happens a few times per week. Additionally I noticed this issue reported here after upgrading to core-2024.11.1 (from core-2024.10.4). But I can't really tell it's related to new core version as I upgraded my dongle as well two weeks ago Best Eric |
|
on core-2024.11.2 also have a SONOFF ZB Dongle-P, have the same error as the OP |
|
@nayefalebrahim please attach a ZHA debug log |
|
I'm seeing the same with a S31 Lite zb by SONOFF. Here's the logs: Here's where it fails in the system logs with the Zigbee debug enabled: |
|
@christabone |
|
@puddly Thank you for the insight on that issue, I'll be sure to check the coordinator! EDIT: Moving it further away with an extension cable fixed the issue, thanks again. |
|
I have only misery with the ZBDongle-E. I now upgraded to the latest ITead version 7.4.4.0 build 0 and will report back if that makes a difference |
|
Hi @puddly I'm now testing with the updated Firmware (iTead 7.4.4) and ZHA ran for almost 2 days without errors or warnings. It was running again without reseating the dongle 🤞 |
|
@puddly This realy an issue in ZHA!
This was found in the log |
|
@erkr Enable debug logging for a few minutes and post the full log when ZHA is in this bad state. |
|
@puddly 👌 |
|
@puddly I added the log ending with disabling the debug logging. Some timestamps:
|
|
Also seeing this issue sometime after upgrading to 2024.11. I'm runnning home assistant in an LXC. I'll try and grab some logs. |
|
This is what I saw. Everything is great, and then suddenly I get a "no such device" error. Looking in the container the device is still there. |
|
@puddly I created an automation that notified me when the last seen's stop updating (logging: Integration not responsive: Check zha)
The log doesn't contain a single ZHA related logging! |
|
Two days ago, something slightly different happened. The symptoms were identical, except that there was 1 error logged:
|
|
I'm going to try and will back to an older version I think and see if this problem goes away. Otherwise perhaps it's time to upgrade my controller. |
|
Ive rolled back to 2024.10.3, I'll update later this week. Grasping at straws, but I wonder if its related to this change? The logic for a lost serial connection has changed. |
|
Little update: as I was desperate I found a configure zigbee button under system/hardware. |
|
I also encountered that ZHA was up and running until I restarted core. After that ZHA failed to initialise |
|
So no luck and I also noticed that as of 2024.12 the backup feature also no longer works. |
|
@puddly I downgrade core back to 10.4 and the issues were resolved. |
|
I'm on 2024.12 with the SONOFF Zigbee 3.0 USB Dongle Plus-E on 7.4.5 firmware and I'm having issues as well. Running on dedicated x86 hardware |
|
@puddly little update here... |
|
Whats interesting is that I can get ZHA to crash almost on command during firmware updates to these new Third Reality plugs I'm installing for better network coverage. |
|
@Ithrial Can you enable debug logging and post a log that includes the crash? |
|
yep - debugging enabled - as soon as my Third Realty plugs prompt me for an update, I'll clear my logs, kick off the update and watch it crash. The update is from firmware 0x10013050 to 0x10013058 Though now that I look at it, the steps to enable the Debug Logging is straight forward but I'm not 100% where to get the logs from unless they're in the main HA logs I can pull from the File Editor. I looked at the steps for adding logging and it said do some steps in the config.yaml file - my config.yaml has hardly anything in it so hopefully I can get some info that'll be helpful |
|
A log will immediately download when you click the "Disable debug logging" button. You can speed up the OTA checks by explicitly clicking the refresh button, it'll ask that all devices check in immediately: |
|
Got it - Update on "Livingroom Zigbee Switch 2" started where it had left off (around 20%) and ran to about 40ish % then crashed. Double checked if I could restart and gave me errors and failure - then I tried just toggling one of the other switches that was within ear shot to hear if it toggled (with up to date firmware) and thats where I get the error "Failed to perform the action switch/turn_off. Failed to send request: ApplicationController is not running" See log |
|
I was away for the holidays and Santa apparently gave me the same bug. Throwing: @ryanwinter I'm also on 2024.10.3 now |
|
Same issue here with sonof dongle P. I replaced this with a connect ZBT-1 and it’s still giving me the same issue. Logger: homeassistant.components.websocket_api.http.connection [281472362328800] Unexpected exception The above exception was the direct cause of the following exception: Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last): |
|
Also getting the following in the logbook Logger: homeassistant.components.websocket_api.http.connection [281472362322608] Error handling message: Unknown error (unknown_error) Peter from 192.168.0.123 (Mozilla/5.0 (iPad; CPU OS 18_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2024.12.2 (io.robbie.HomeAssistant; build:2024.1058; iPadOS 18.2.0) Mobile/HomeAssistant, like Safari) |
|
Attached is a debug log what I also noticed is that when I unplug the controller and plug it back in or when I do a full restart of the home assistant green device the zigbee switches work for about 10 minutes after which none of my zigbee devices are working anymore until I do the same thing again. This makes home assistant useless for me at this point as none of my automations that I need are working anymore. |
|
@petepall Can you enable ZHA debug logging, let it run for however long it takes to get How are you running Home Assistant? Docker? A VM? |
Hi puddly, my Zhang’s is not recovering anymore until I disconnect and connect the device or do a full system reboot. |
|
I tested connecting the controller directly to the home assistant green (with extension cable) and using a powered usb hub. Issue happens in both cases. |
|
Here is a new log that spans from the device working to it getting disabled and working again. |
|
@Ithrial It looks like the firmware on your stick partially crashes during OTA and the stick refuses to acknowledge any data sent to it. At the same time, it isn't hearing back any acknowledgements from ZHA. It's as if the serial connection's TX line disconnects at some point. This is all only happening when there is a lot of data flowing back and forth (i.e. OTA + controlling a device at the same time). I also see that you're running a firmware build with hardware flow control disabled. Can you try an alternative firmware for it that's configured like the one we use for the SkyConnect? sonoff-e_zigbee_ncp_hw_flow_7.4.4.0.gbl.zip (make sure to unzip it first). You can flash it the same way you probably upgraded your stick originally (via the web flasher), just pick this GBL file. Once that's done, re-configure ZHA to change the serial port's flow control to |
|
@petepall I see you're also running EmberZNet 7.4.4.0 on your ZBT-1. Do you remember when you installed it? How did you do so? |
|
@puddly I received the stick with this software on it. I today unboxed it and start using it as I thought the issue was with my sonoff dongle P |
|
@puddly - Thank you for looking at the logs. I'm installing the firmware that you linked now and I have some firmware updates needed on the plugs - will change to hardware flow control and reply back |
|
@puddly First follow-up: Hardware: reconfigure with hardware flow control hung on initializing. Rebooted HA after ZHA hung at Initializing. ZHA started and everything is detected but now the Third Reality plugs are showing as up to date, even after a manual updates check. Can confirm network is up by toggling the TR Plugs power on and off. Will report back with next firmware update |
|
@puddly: 2nd Follow-up: Just got the prompt for my 4x Third Reality Zigbee Plugs. I updated the 1st plug thats closest to the Coordinator. Completely 100% successfully with 0 crashes. I forgot to enable Debugging mode so thankfully it completed on the first click. I enabled Debugging mode and continued with the next closest which is about 12-15ft away. in the Kitchen - it too completed 100% in the first go. Next I tried upgrading the 2 farthest away (both equidistant from the Kitchen Plug). They're slowly but surely updating (update much slower when multiple devices are updating at the same time which makes sense in a network with limited bandwidth). The update of one of the 2 simultaneous updates did crash around 80ish percent, however I was able to resume it without having to restart HA, which is a first. The 2nd of the 2 simultaneous updates completed successfully with no issues. I'm not sure if hardware data flow control is enabled or no but I'm perfectly content in the state that its in right now, from a stability perspective The log file from the 3 updates is 155MB for some reason so it wont let me upload the thread so I had to upload to Google Drive - See Log |
|
Thanks! So as far as I can tell, the firmware doesn't crash at all once you use hardware flow control. OTA failing with concurrent updates is a different issue but I think we have the firmware problem figured out. What hardware and software are you running Home Assistant OS/container on? @petepall The ZBT-1 shouldn't ship with 7.4.4.0, it should come with 7.3.1.0. Was your ZBT-1 used? If not, which distributor did you buy it from? |
|
@puddly Thanks for looking at the logs. I'm running the HAOS via the generic x86 install options on a dedicated N97 based mini pc. |
|
@puddly the ZBT-1 was not used as far as I know. I bought it from HA shop. Today I changed again the extension cable and removed my powered usb hub and till now the devices are now working. |
|
@petepall Can you link to the store? |
Here is the link https://www.hashop.nl/Home-Assistant-SkyConnect#ProductReviewText |
The problem
My ZHA entities couldn’t be controlled anymore although all ZHA entities seemed available and fine. No sensor updates and errors logged when sending commands (example logging included was a turn on service) .
Even reloading ZHA suggested that ZHA was fine, but the failure state remained. Shortly Un-plugging the ZBDongle-E (Sonoff stock Firmware: 6.10.3.0 build 297), resolved the issue.
So for some reason the ZHA watchdog didn’t notice this hanging dongle state for hours!
What version of Home Assistant Core has the issue?
Core-2024.11.1
What was the last working version of Home Assistant Core?
No response
What type of installation are you running?
Home Assistant OS
Integration causing the issue
ZHA
Link to integration documentation on our website
https://www.home-assistant.io/integrations/zha/
Diagnostics information
No response
Example YAML snippet
No response
Anything in the logs that might be useful for us?
Logger: homeassistant.components.websocket_api.http.connection Source: components/websocket_api/commands.py:245 integration: Home Assistant WebSocket API (documentation, issues) First occurred: 19:14:03 (2 occurrences) Last logged: 19:14:08 [546888653200] Unexpected exception Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/__init__.py", line 67, in wrap_zigpy_exceptions yield File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/__init__.py", line 85, in wrapper return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zigpy/util.py", line 136, in retry return await func() ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zigpy/zcl/__init__.py", line 375, in request return await self._endpoint.request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zigpy/endpoint.py", line 270, in request return await self.device.request( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zigpy/device.py", line 378, in request await send_request() File "/usr/local/lib/python3.12/site-packages/zigpy/application.py", line 841, in request await self.send_packet( File "/usr/local/lib/python3.12/site-packages/bellows/zigbee/application.py", line 700, in send_packet raise ControllerError("ApplicationController is not running") bellows.exception.ControllerError: ApplicationController is not running The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1335, in handler return await func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/zha/switch.py", line 57, in async_turn_on await self.entity_data.entity.async_turn_on() File "/usr/local/lib/python3.12/site-packages/zha/application/platforms/switch.py", line 95, in async_turn_on await self._on_off_cluster_handler.turn_on() File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/general.py", line 465, in turn_on result = await self.on() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/__init__.py", line 84, in wrapper with wrap_zigpy_exceptions(): File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/__init__.py", line 76, in wrap_zigpy_exceptions raise ZHAException(message) from exc zha.exceptions.ZHAException: Failed to send request: ApplicationController is not running The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service response = await hass.services.async_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2802, in async_call response_data = await coro ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2845, in _execute_service return await target(service_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1007, in entity_service_call single_response = await _handle_entity_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1079, in _handle_entity_call result = await task ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 628, in async_handle_light_on_service await light.async_turn_on(**filter_turn_on_params(light, params)) File "/usr/src/homeassistant/homeassistant/components/switch_as_x/entity.py", line 152, in async_turn_on await self.hass.services.async_call( File "/usr/src/homeassistant/homeassistant/core.py", line 2802, in async_call response_data = await coro ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2845, in _execute_service return await target(service_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1007, in entity_service_call single_response = await _handle_entity_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1079, in _handle_entity_call result = await task ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1337, in handler raise HomeAssistantError(err) from err homeassistant.exceptions.HomeAssistantError: Failed to send request: ApplicationController is not runningAdditional information
All last seen timestamps where not updated (3 hours old when I noticed). So all mains devices should have been made unavailable by ZHA (my timeout for mains devices is 5 minutes)
The text was updated successfully, but these errors were encountered: