Upgrade to IDF 5.3.2 #3664

jmattsson · 2024-10-17T05:08:06Z

This PR is for the dev-esp32 branch rather than for the release branch.
This PR is compliant with the other contributing guidelines as well (if not, please describe why).
I have thoroughly tested my contribution.
[n/a] The code changes are reflected in the documentation at docs/*.

The core of this PR is the upgrade of the ESP IDF to the latest, 5.3.2. There were only a few adjustments required to make NodeMCU build with it, and a couple more to make it run. Specifically, the VFS console driver needed to be explicitly initialised now, and our existing issue with WiFi+USB-CDC had a regression which necessitated a change to the workaround. The workaround is only enabled if USB-CDC console is used.

A second issue was discovered on the ESP32-S2 with USB-CDC enabled, whereby flashing of a blank unit failed to start up. After quite a bit of digging I narrowed this down to the auto-format of the SPIFFS partition. That much flash access seems to interfere with the USB device enumeration, and results in a failed boot / the unit being repeatedly reset by the USB host. When the USB-CDC console is enabled, we now default to not auto-formatting, though this can be overridden via the new Kconfig option.

While poking around in this area I also discovered another pre-existing issue where if the initial SPIFFS mount failed, then a file.format() would not mount the newly formatted filesystem, necessitating a node.restart() before it was usable. This has now been addressed by requesting a (re)mount after a successful format, giving consistent behaviour in both cases. As part of fixing this and the previous issue, the default filesystem mounting code has been moved into our platform component, as it's needed both by user_main.c and the file module.

Basic testing has been done on: ESP32, ESP32-C3, ESP32-C6, ESP32-H2 and ESP32-S2.
Additional testing by others is welcome and encouraged.

Edit: Point upgrade from 5.3.1 to 5.3.2 now also included, but less tested.

serg3295 · 2024-10-17T14:18:10Z

I have tested the PR on esp32, esp32-c3 super mini, nanoESP32-C6 v1.0, esp32-s3 zero, esp32-s2 Mini v1.0.0 for correct operation with nodemcu-tool utility and nodemcu-tools VSCode extension.

all connections via USB

esp32
port: UART0 - works as expected.

esp32c3 and esp32c6
port: usb-serial-jtag
behavior =>
uart.write(0) causes 'uart driver error' message
using
uart.start(0) -- no 'driver error' message, but
uart.write(0,'a\n') -- nothing is sent to the console

Applying changes as described in issue #3657 solves the problem. Both utilities work fine.

nano esp32-c6
port: UART0 (ch343) -- works as expected without changing "platform.c"

esp32-s3 zero
Chip is ESP32-S3 (QFN56) (revision v0.1)
Features: WiFi, BLE, Embedded Flash 4MB (XMC), Embedded PSRAM 2MB (AP_3v3)

port: usb-serial-jtag
idf5.1.3 REPL -> O.K.
idf5.2.2 REPL -> O.K.
idf5.2.3 REPL, but typing any char in console reboots esp
idf5.3.1 repeated reboot

esp32-s2 Mini
Chip is ESP32-S2FNR2 (revision v0.0)
Features: WiFi, Embedded Flash 4MB, Embedded PSRAM 2MB, ADC and temperature sensor calibration in BLK2 of efuse V2
port: USB-CDC
behavior =>
I have only REPL in console (make monitor /dev/ttyACM0). Both utilities does not work.
Sending simple print("echo1337") sometimes causes

E (295) mmap: esp_mmu_paddr_to_vaddr(805): paddr isn't mIprint("echo1337")

Wifi errors

the same issue #3660 is on esp32c3 and esp32c6 with idf5.3.1

serg3295 · 2024-10-17T16:21:42Z

Fixes warnings

diff --git a/components/base_nodemcu/user_main.c b/components/base_nodemcu/user_main.c
index 82453afd..f2391b22 100644
--- a/components/base_nodemcu/user_main.c
+++ b/components/base_nodemcu/user_main.c
@@ -24,6 +24,7 @@
 #include "esp_vfs_usb_serial_jtag.h"
 #include "driver/uart_vfs.h"
 #include "driver/usb_serial_jtag.h"
+#include "driver/usb_serial_jtag_vfs.h"
 #include "nvs_flash.h"
 
 #include "task/task.h"
@@ -240,15 +241,15 @@ static void console_init(void)
 #elif CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG
   /* Based on @pjsg's work */
 
-  esp_vfs_dev_usb_serial_jtag_set_rx_line_endings(RX_LINE_ENDINGS_CFG);
-  esp_vfs_dev_usb_serial_jtag_set_tx_line_endings(TX_LINE_ENDINGS_CFG);
+  usb_serial_jtag_vfs_set_rx_line_endings(RX_LINE_ENDINGS_CFG);
+  usb_serial_jtag_vfs_set_tx_line_endings(TX_LINE_ENDINGS_CFG);
 
   usb_serial_jtag_driver_config_t usb_serial_jtag_config =
     USB_SERIAL_JTAG_DRIVER_CONFIG_DEFAULT();
   /* Install USB-SERIAL-JTAG driver for interrupt-driven reads and write */
   usb_serial_jtag_driver_install(&usb_serial_jtag_config);
 
-  esp_vfs_usb_serial_jtag_use_driver();
+  usb_serial_jtag_vfs_use_driver();
 #elif CONFIG_ESP_CONSOLE_USB_CDC
   /* Based on console/advanced_usb_cdc */

jmattsson · 2024-10-18T00:46:52Z

@serg3295 Oooh, thank you for testing!

Let's see what we can do about these (and what I can reproduce with what kit I have available)

jmattsson · 2024-10-18T03:48:19Z

esp32c3 and esp32c6
port: usb-serial-jtag
behavior =>
uart.write(0) causes 'uart driver error' message
using
uart.start(0) -- no 'driver error' message, but
uart.write(0,'a\n') -- nothing is sent to the console

This is correct behaviour. If you select the USB-SERIAL-JTAG as the console, writing to UART0 will not go to the console. These are two separate hardware devices. USB-SERIAL-JTAG is not a UART. If you hook up an external UART-to-USB adapter to the UART0 pins you should see the "a\n" appear there.

esp32-s3 zero
Chip is ESP32-S3 (QFN56) (revision v0.1)
Features: WiFi, BLE, Embedded Flash 4MB (XMC), Embedded PSRAM 2MB (AP_3v3)

port: usb-serial-jtag
idf5.1.3 REPL -> O.K.
idf5.2.2 REPL -> O.K.
idf5.2.3 REPL, but typing any char in console reboots esp
idf5.3.1 repeated reboot

I don't currently have this dev board (I've ordered one now). Could you please post the crash log from make monitor on 5.3.1 here, and I can see if I can figure it out without hardware?

serg3295 · 2024-10-18T06:07:59Z

Could you please post the crash log from make monitor on 5.3.1 here, and I can see if I can figure it out without hardware?

Crash log

esp32-S3 zero usb-serial-jtag

Serial port /dev/ttyACM0
Connecting...
Detecting chip type... ESP32-S3
Running idf_monitor in directory /home/serg/Projects/lua/nodeMCU-firmware
Executing "/home/serg/.espressif/python_env/idf5.3_py3.12_env/bin/python /home/serg/Projects/lua/nodeMCU-firmware/sdk/esp32-esp-idf/tools/idf_monitor.py -p /dev/ttyACM0 -b 115200 --toolchain-prefix riscv32-esp-elf- --target esp32c3 --revision 3 --decode-panic backtrace /home/serg/Projects/lua/nodeMCU-firmware/build/nodemcu.elf -m '/home/serg/.espressif/python_env/idf5.3_py3.12_env/bin/python' '/home/serg/Projects/lua/nodeMCU-firmware/sdk/esp32-esp-idf/tools/idf.py'"...
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x40048dfc
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce2810,len:0x178c
load:0x403c8700,len:0x4
load:0x403c8704,len:0xc10
load:0x403cb700,len:0x2dc0
entry 0x403c8904
�[0;32mI (26) boot: ESP-IDF v5.3.1 2nd stage bootloader�[0m
�[0;32mI (27) boot: compile time Oct 17 2024 17:48:31�[0m
�[0;32mI (27) boot: Multicore bootloader�[0m
�[0;32mI (27) boot: chip revision: v0.1�[0m
�[0;32mI (27) boot.esp32s3: Boot SPI Speed : 80MHz�[0m
�[0;32mI (27) boot.esp32s3: SPI Mode       : DIO�[0m
�[0;32mI (28) boot.esp32s3: SPI Flash Size : 4MB�[0m
�[0;32mI (28) boot: Enabling RNG early entropy source...�[0m
�[0;32mI (28) boot: Partition Table:�[0m
�[0;32mI (28) boot: ## Label            Usage          Type ST Offset   Length�[0m
�[0;32mI (29) boot:  0 nvs              WiFi data        01 02 00009000 00006000�[0m
�[0;32mI (29) boot:  1 phy_init         RF data          01 01 0000f000 00001000�[0m
�[0;32mI (29) boot:  2 factory          factory app      00 00 00010000 00180000�[0m
�[0;32mI (30) boot:  3 lfs              unknown          c2 01 00190000 00010000�[0m
�[0;32mI (30) boot:  4 storage          Unknown data     01 82 001a0000 00070000�[0m
�[0;32mI (31) boot: End of partition table�[0m
�[0;32mI (31) esp_image: segment 0: paddr=00010020 vaddr=3c0c0020 size=2d5e8h (185832) map�[0m
�[0;32mI (65) esp_image: segment 1: paddr=0003d610 vaddr=3fc9ab00 size=02a08h ( 10760) load�[0m
�[0;32mI (67) esp_image: segment 2: paddr=00040020 vaddr=42000020 size=bf524h (783652) map�[0m
�[0;32mI (208) esp_image: segment 3: paddr=000ff54c vaddr=3fc9d508 size=01f5ch (  8028) load�[0m
�[0;32mI (210) esp_image: segment 4: paddr=001014b0 vaddr=40374000 size=16a20h ( 92704) load�[0m
�[0;32mI (231) esp_image: segment 5: paddr=00117ed8 vaddr=600fe000 size=00100h (   256) load�[0m
�[0;32mI (231) esp_image: segment 6: paddr=00117fe0 vaddr=600fe100 size=00008h (     8) load�[0m
�[0;32mI (241) boot: Loaded app from partition at offset 0x10000�[0m
�[0;32mI (241) boot: Disabling RNG early entropy source...�[0m
�[0;32mI (242) cpu_start: Multicore app�[0m
�[0;32mI (251) cpu_start: Pro cpu start user code�[0m
�[0;32mI (251) cpu_start: cpu freq: 160000000 Hz�[0m
�[0;32mI (251) app_init: Application information:�[0m
�[0;32mI (252) app_init: Project name:     nodemcu�[0m
�[0;32mI (252) app_init: App version:      tmr-libmain-binpatch150-891-gb7�[0m
�[0;32mI (252) app_init: Compile time:     Oct 17 2024 17:48:28�[0m
�[0;32mI (252) app_init: ELF file SHA256:  5cc304fc0...�[0m
�[0;32mI (252) app_init: ESP-IDF:          v5.3.1�[0m
�[0;32mI (253) efuse_init: Min chip rev:     v0.0�[0m
�[0;32mI (253) efuse_init: Max chip rev:     v0.99 �[0m
�[0;32mI (253) efuse_init: Chip rev:         v0.1�[0m
�[0;32mI (253) heap_init: Initializing. RAM available for dynamic allocation:�[0m
�[0;32mI (253) heap_init: At 3FCA3A48 len 00045CC8 (279 KiB): RAM�[0m
�[0;32mI (254) heap_init: At 3FCE9710 len 00005724 (21 KiB): RAM�[0m
�[0;32mI (254) heap_init: At 3FCF0000 len 00008000 (32 KiB): DRAM�[0m
�[0;32mI (254) heap_init: At 600FE108 len 00001EE0 (7 KiB): RTCRAM�[0m
�[0;32mI (256) spi_flash: detected chip: generic�[0m
�[0;32mI (256) spi_flash: flash io: dio�[0m
�[0;33mW (256) rmt(legacy): legacy driver is deprecated, please migrate to `driver/rmt_tx.h` and/or `driver/rmt_rx.h`�[0m
�[0;33mW (256) i2c: This driver is an old driver, please migrate your application code to adapt `driver/i2c_master.h`�[0m
�[0;33mW (257) ADC: legacy driver is deprecated, please migrate to `esp_adc/adc_oneshot.h`�[0m
�[0;32mI (257) sleep: Configure to isolate all GPIO pins in sleep state�[0m
�[0;32mI (258) sleep: Enable automatic switching of GPIO sleep configuration�[0m
�[0;32mI (258) main_task: Started on CPU0�[0m
�[0;32mI (268) main_task: Calling app_main()�[0m

***ERROR*** A stack overflow in task console has been detected.


Backtrace: 0x40375c0e:0x3fcae220 0x4037e865:0x3fcae240 0x4037f572:0x3fcae260 0x40380aa7:0x3fcae2e0 0x4037f6a4:0x3fcae310 0x4037f69a:0x3fcae2fc |<-CORRUPTED



ELF file SHA256: 5cc304fc0

Rebooting...
ESP-ROM:esp32s3-20210327

debugger log

Set GDB target to 'esp32s3.cpu0'
2
[esp32s3.cpu1] Target halted, PC=0x4037BE8A, debug_reason=00000000
Set GDB target to 'esp32s3.cpu1'
[esp32s3.cpu0] Target halted, PC=0x40375C11, debug_reason=00000001
[esp32s3.cpu0] Halt cause (***ERROR*** A stack overflow in task console has been detected.)
[New Thread 1070261884]
[New Thread 1070259388]
[New Thread 1070251056]

Thread 9 "console" received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 1070261884]
0x40375c11 in panic_abort (details=0x3fcae1a0 "***ERROR*** A stack overflow in task console has been detected.") at /home/serg/Projects/lua/nodeMCU-firmware/sdk/esp32-esp-idf/components/esp_system/panic.c:463
463	    *((volatile int *) 0) = 0; // NOLINT(clang-analyzer-core.NullDereference) should be an invalid operation on targets

Solution

I have increased console_task stack to 1152 and the error has gone.

  xTaskCreate(
    console_task, "console", 1152, NULL, ESP_TASK_MAIN_PRIO+1, NULL);
}

jmattsson · 2024-10-18T07:08:54Z

Thanks @serg3295 !

Alright, this is interesting. I was chasing down why I needed the explicit call to esp_vfs_console_register() when by all evidence it should not be needed because it's supposed to get automatically called from the low-level startup code. Yet without it I kept seeing crashes on ESP32 as soon as a character was read.

To my surprise, when I increased the stack size for the console task this issue too disappeared. Hmm. I'm not sure whether this makes me happy or deeply unhappy... I didn't have either of the stack overflow check methods complain for me.

serg3295 · 2024-10-18T11:06:34Z

If you hook up an external UART-to-USB adapter to the UART0 pins you should see the "a\n" appear there.

I see. Thanks. I received a message to the UART0 connected via the FTDI adapter

If I understand correctly, the only way to send a message to the CDC and JTAG console from esp is 'print()'?

jmattsson · 2024-12-09T07:27:19Z

Announcing intent to merge despite lack of peer review — I'd rather be told off than have the project stagnate into obsolescence 🙃

serg3295 · 2024-12-09T09:57:09Z

I merged the dev-esp32-console-module branch into this PR and notice that esp_vfs_cdcacm.h was moved into
/sdk/esp32-esp-idf/components/esp_vfs_console/include/esp_vfs_cdcacm.h

After fixing the path, the firmware was compiled and works fine.

To help compile-test the different console options, for starters.

jmattsson · 2024-12-10T02:21:27Z

I had to fix up some additional deprecation warnings in the console module, but that should all be sorted. I've also updated the CI build to actually build with different console options, for better regression check coverage.

Of course, IDF 5.3.2 was released earlier, so maybe I should look at updating this PR to go straight to that?

pjsg · 2024-12-10T04:07:12Z

Thanks for this PR -- I merged it into my PR to get to IDF 5.5 for the ESP32C5 support. I'm left with having to fixup the CAN support which doesn't build on 5.5 anyway (and I need it for the C5) Philip Gladstone [image: Bitsight website] <https://pskreporter.info> [image: Bitsighttwitter] <https://twitter.com/n1dq>

…

On Mon, Dec 9, 2024 at 9:22 PM Jade Mattsson ***@***.***> wrote: I had to fix up some additional deprecation warnings in the console module, but that should all be sorted. I've also updated the CI build to actually build with different console options, for better regression check coverage. Of course, IDF 5.3.2 was released earlier, so maybe I should look at updating this PR to go straight to that? — Reply to this email directly, view it on GitHub <#3664 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALQLTIEPJNSPBR2NC3IJHT2EZF33AVCNFSM6AAAAABQC4GTYCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZQGA3DGOJUGA> . You are receiving this because your review was requested.Message ID: ***@***.***>

Having gone through the changelog from 5.3.1, this looks safe enough...

jmattsson added enhancement ESP32 labels Oct 17, 2024

jmattsson requested a review from pjsg October 17, 2024 05:08

Upgrade IDF to 5.3.1

4e6b842

jmattsson force-pushed the dev-esp32-idf5.3.1 branch from 4931b37 to 4e6b842 Compare December 10, 2024 01:31

Add chip-specific build defaults to CI

134f21d

To help compile-test the different console options, for starters.

jmattsson force-pushed the dev-esp32-idf5.3.1 branch from e0725f5 to 134f21d Compare December 10, 2024 02:11

Update IDF to v5.3.2

be6f6c5

Having gone through the changelog from 5.3.1, this looks safe enough...

jmattsson changed the title ~~Upgrade to IDF 5.3.1~~ Upgrade to IDF 5.3.2 Dec 10, 2024

jmattsson merged commit 3dc3dac into nodemcu:dev-esp32 Dec 18, 2024
24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to IDF 5.3.2 #3664

Upgrade to IDF 5.3.2 #3664

jmattsson commented Oct 17, 2024 •

edited

Loading

serg3295 commented Oct 17, 2024

serg3295 commented Oct 17, 2024

jmattsson commented Oct 18, 2024

jmattsson commented Oct 18, 2024

serg3295 commented Oct 18, 2024

jmattsson commented Oct 18, 2024

serg3295 commented Oct 18, 2024 •

edited

Loading

jmattsson commented Dec 9, 2024

serg3295 commented Dec 9, 2024

jmattsson commented Dec 10, 2024

pjsg commented Dec 10, 2024 via email

Upgrade to IDF 5.3.2 #3664

Upgrade to IDF 5.3.2 #3664

Conversation

jmattsson commented Oct 17, 2024 • edited Loading

serg3295 commented Oct 17, 2024

all connections via USB

Wifi errors

serg3295 commented Oct 17, 2024

jmattsson commented Oct 18, 2024

jmattsson commented Oct 18, 2024

serg3295 commented Oct 18, 2024

Solution

jmattsson commented Oct 18, 2024

serg3295 commented Oct 18, 2024 • edited Loading

jmattsson commented Dec 9, 2024

serg3295 commented Dec 9, 2024

jmattsson commented Dec 10, 2024

pjsg commented Dec 10, 2024 via email

jmattsson commented Oct 17, 2024 •

edited

Loading

serg3295 commented Oct 18, 2024 •

edited

Loading