guild icon
Toit
#Watchdogs & jag monitor
Thread channel in help
addshore
addshore 03/27/2025 09:30 AM
Im playing around with watchdogs for one of the first times in toit.
When the watchdog times out, jag monitor disconnects (probably fine) however I stop seeing any output from the ESP, when reconnecting, even though my container is still running / has restarted

[watchdog] ERROR: watchdog too late {id: lightbug/otl/link} [toit] INFO: entering deep sleep for 50ms Error: Port has been closed ~/dev/lb/io/toit (main) jag monitor --attach Starting serial monitor of port '/dev/ttyACM0' ...

What (if anything) am I doing wrong?
addshore
addshore 03/27/2025 09:32 AM
Oh intesting, when re installing the container, I then get this

Starting serial monitor of port '/dev/ttyACM0' ... (234138) task_wdt: - sys_evt (CPU 0) E (234138) task_wdt: Tasks currently running: E (234138) task_wdt: CPU 0: IDLE E (234138) task_wdt: Aborting. Core 0 register dump: MEPC : 0x40811dfc RA : 0x4211bb34 SP : 0x408303c0 GP : 0x40823204 TP : 0x40830400 T0 : 0x00000000 T1 : 0x40811b82 T2 : 0x00000000 S0/FP : 0x4082c000 S1 : 0x40826674 A0 : 0x00000000 A1 : 0x00000000 A2 : 0x00000000 A3 : 0x00000000 A4 : 0x00000000 A5 : 0x00000000 A6 : 0x00000000 A7 : 0x00000000 S2 : 0x4082c000 S3 : 0x00000000 S4 : 0x00000000 S5 : 0x00000000 S6 : 0x00000000 S7 : 0x00000000 S8 : 0x00000000 S9 : 0x00000000 S10 : 0x00000000 S11 : 0x00000000 T3 : 0x00000000 T4 : 0x00000000 T5 : 0x00000000 T6 : 0x00000000 MSTATUS : 0x00000000 MTVEC : 0x00000000 MCAUSE : 0x00000000 MTVAL : 0x00000000 MHARTID : 0x00000000

And the esp restarts
bitphlipphar
bitphlipphar 03/27/2025 03:03 PM
If you don't feed the watchdog in time, the system will restart.
bitphlipphar
bitphlipphar 03/27/2025 03:03 PM
But it is a little weird that you lose the serial monitoring in the process. Is this on a c6?
addshore
addshore 03/28/2025 08:57 AM
yup, on a c6!
addshore
addshore 03/28/2025 08:57 AM
It touches on another thing I wanted to look at in jag actually, such as making the watch command have a --reattach argument for example
addshore
addshore 03/28/2025 09:13 AM
unrelated, i'd love to add the concept of "wifi profiles" that can remember multiple creds for the flash command, so I dont have to change them up when im moving locations :😄:
👍1
addshoreOPaddshore
It touches on another thing I wanted to look at in jag actually, such as making the watch command have a --reattach argument for example
bitphlipphar
bitphlipphar 03/28/2025 09:18 AM
I think the serial monitoring only disconnects due to resets on some devices. I seem to remember that it is related to something about how much of the UART support is done in software versus hardware. I've had devices where very early crashes in the software stack made it incredibly hard to use the UART for flashing, because the software side of things would get interrupted by the crashes.
bitphlipphar
bitphlipphar 03/28/2025 09:19 AM
Not entirely sure about the c6, but I think I've struggled with an s3 for a few unfun days.
addshoreOPaddshore
Oh intesting, when re installing the container, I then get this ``` Starting serial monitor of port '/dev/ttyACM0' ... (234138) task_wdt: - sys_evt (CPU 0) E (234138) task_wdt: T...
bitphlipphar
bitphlipphar 03/28/2025 09:20 AM
Is this because you can't keep feeding the watchdog from a container that has been shut down? And there is no cleaning up happening at shutdown that stops the watchdog?
bitphlippharbitphlipphar
Is this because you can't keep feeding the watchdog from a container that has been shut down? And there is no cleaning up happening at shutdown that stops the watchdog?
addshore
addshore 03/28/2025 09:21 AM
potentially,
I saw the advise that I should be running the watchdogs in a seperate container (which I am not currently doing) so likely I just need to use the watchdogs better
bitphlipphar
bitphlipphar 03/28/2025 09:35 AM
The high-level watchdog support is built on top of a low-level watchdog provided by ESP-IDF. We feed the low-level watchdog if the high-level watchdog are properly fed and we've noticed it. If the service provider runs along side other software it may be starved, which will lead to the low-level watchdog not being fed even though you've done your best to feed the high-level ones.
bitphlipphar
bitphlipphar 03/28/2025 09:35 AM
That is why we recommend running the low-level feeding provider separately, so the likelihood of that happening is low.
bitphlipphar
bitphlipphar 03/28/2025 09:37 AM
The loop that feeds the low-level watchdog - unless we've noticed a high-level watchdog that hasn't been fed - is here: https://github.com/toitware/toit-watchdog/blob/main/src/provider.toit#L111
addshore
addshore 03/28/2025 11:29 AM
re jag monitor, this is another one I get, where I just can't get it to close if something has gone wrong

jag monitor Starting serial monitor of port '/dev/ttyACM0' ... ^C^C
addshore
addshore 03/28/2025 11:46 AM
killing it even seems to take some crazy ammount of time
16 messages in total