Guru Meditation Error

Toit

#Guru Meditation Error

Thread channel in help

RobvanLopik 09/18/2024 08:23 PM

what is it? I am letting Jaguar run a container that deepsleeps and I experiance all kinds of failures.

OPRobvanLopik

what is it? I am letting Jaguar run a container that deepsleeps and I experiance all kinds of failures.

floitsch 09/18/2024 08:38 PM

Can you share the output you are seeing?

floitsch 09/18/2024 08:38 PM

Guru meditation errors are crashes in the system.

floitsch 09/18/2024 08:38 PM

Normally you should never see these.

floitsch 09/18/2024 08:40 PM

Which chip (esp32, esp32s3, ...) are you using?
Which version of Jaguar? (jag version).
Can you share a reproducible test-case? (but maybe share the full output first, before you spend resources creating the repro).

RobvanLopik 09/18/2024 09:07 PM

using version jag 1.41.0 on 3 yr old ESP32 - revision 3.0. Part of the output:

RobvanLopik 09/18/2024 09:09 PM

time to sleep
Guru Meditation Error: Core 0 panic'ed (InstrFetchProhibited). Exception was unhandled.

Core 0 register dump:
PC : 0x00000000 PS : 0x00060430 A0 : 0x8011f1e7 A1 : 0x3ffd7c80
A2 : 0x3ffe07a8 A3 : 0x3ffe2710 A4 : 0x0000000c A5 : 0x3ffbbd9c
A6 : 0x00000000 A7 : 0x00000000 A8 : 0x801b90d4 A9 : 0x3ffd7c40
A10 : 0x00000000 A11 : 0x3ffe2710 A12 : 0x3ffd5f2c A13 : 0x00000000
A14 : 0x00000000 A15 : 0x3ffd5ed8 SAR : 0x00000002 EXCCAUSE: 0x00000014
EXCVADDR: 0x00000000 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0xffffffff

Decoding by jag, device has version <2.0.0-alpha.160>

Backtrace: 0xfffffffd:0x3ffd7c80 0x4011f1e4:0x3ffd7ca0 0x4010ec37:0x3ffd7cc0 0x400dce23:0x3ffd7ce0 0x400dcee5:0x3ffd7d00 0x400e261c:0x3ffd7d20 0x4010e0cd:0x3ffd7d50
jag: Failed to decode line.

**`**

`ELF file SHA256: 8e703947df2618a5

Rebooting...
ets Jul 29 2019 12:21:46

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12700
ho 0 tail 12 room 4
load:0x40080400,len:2916
entry 0x400805c4
[toit] INFO: starting <v2.0.0-alpha.160>
[toit] INFO: running on ESP32 - revision 3.0
[jaguar``

floitsch 09/18/2024 09:11 PM

You might be hitting this issue: https://github.com/toitlang/toit/issues/1390

Crash when closing socket with unconsumed input after network is cl...

When closing TCP and UDP sockets on the ESP32, we call pbuf_free if we hold onto network packets that we got from the underlying network layer (like wifi). This in return calls back into the networ...

floitsch 09/18/2024 09:13 PM

Let me try to decode the stacktrace you received.

floitsch 09/18/2024 09:17 PM

Yep. Looks like it's the same issue:

0xfffffffd: _rtc_slow_reserved_end + 0xafffdffd
0x4011f1e4: esp_pbuf_free + 0xc
0x4010ec37: pbuf_free + 0x2f
0x400dce23: toit::LwipSocket::tear_down() + 0x53
0x400dcee5: std::_Function_handler<toit::Object* (), toit::SocketResourceGroup::on_unregister_resource(toit::Resource*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) + 0x9
0x400e261c: toit::LwipEventSource::on_thread(void*) + 0x14
0x4010e0cd: tcpip_thread + 0xa9

floitsch 09/18/2024 09:17 PM

@bitphlipphar

has a better idea of what exactly the problem is. Maybe he has an easy work-around.

floitsch 09/18/2024 09:19 PM

I think one way to avoid the crash is to close down the network by hand before calling deep-sleep. If that's possible.

RobvanLopik 09/18/2024 09:20 PM

in my code the deepsleep occurs after printing "time to sleep"

RobvanLopik 09/18/2024 09:21 PM

Fortunately the device reboots after this error.

floitsch 09/18/2024 09:21 PM

So from what I understand, it's the shutting down of the containers that's causing this.

RobvanLopik 09/18/2024 09:21 PM

I also saw:
Error: unexpected EOF

floitsch 09/18/2024 09:21 PM

If there is a network socket that has unread data, it might crash the system.

RobvanLopik 09/18/2024 09:21 PM

after which the monitor output stops.

floitsch 09/18/2024 09:22 PM

So there is another issue with "Error: unexpected EOF" and the monitor stopping?

RobvanLopik 09/18/2024 09:23 PM

Yes, and the program continues its deepsleep cycle

floitsch 09/18/2024 09:23 PM

The "Error: unexpected EOF" is printed in the terminal that has the jag monitor running?

RobvanLopik 09/18/2024 09:23 PM

Yes

floitsch 09/18/2024 09:23 PM

and jag monitor keeps running, but there is no new messages anymore?

floitsch 09/18/2024 09:23 PM

even after the device reboots?

floitsch 09/18/2024 09:24 PM

Which platform are you on? (Windows, macOS, Linux)

RobvanLopik 09/18/2024 09:25 PM

No, jag stops

OPRobvanLopik

Yes, and the program continues its deepsleep cycle

floitsch 09/18/2024 09:25 PM

To avoid any confusions: what do you mean with "cycle"?

RobvanLopik 09/18/2024 09:25 PM

I'm on Windows

RobvanLopik 09/18/2024 09:26 PM

It is a simple program that starts an MQTT client, does a measuremnt and publishes the result and the ngoes to deep sleep

floitsch 09/18/2024 09:27 PM

I'm trying to summarize:
- you have jag monitor running.
- you send a container that triggers deepsleep
- at some point (immediately?) the jag monitor program reports "Error: unexpected EOF" and terminates.
- the esp32 however continues working. (observed by seeing it publish something?)

Is this correct?

RobvanLopik 09/18/2024 09:30 PM

yes, the error is not immediate

floitsch 09/18/2024 09:30 PM

interesting.
This might be a different behavior on Windows.

floitsch 09/18/2024 09:31 PM

We are hitting this line: https://github.com/toitlang/jaguar/blob/654f9f5871e60f18beb6c1896c8d197069a3c569/cmd/jag/commands/monitor.go#L124

jaguar/cmd/jag/commands/monitor.go at 654f9f5871e60f18beb6c1896c8d1...

Use live reloading over WiFI to turbo-charge developing for your ESP32. - toitlang/jaguar

RobvanLopik 09/18/2024 09:31 PM

OK, tomorrow afternoon I will try on Linux

floitsch 09/18/2024 09:31 PM

It basically says that if a port read returns 0 bytes (but no error) we got an unexpected EOF.

floitsch 09/18/2024 09:31 PM

Maybe Windows has a timeout that returns from the Read even if there is no data.

floitsch 09/18/2024 09:32 PM

The documentation of the Read function (from the golang package) claims that it should not return with 0 bytes:

// Stores data received from the serial port into the provided byte array
    // buffer. The function returns the number of bytes read.
    //
    // The Read function blocks until (at least) one byte is received from
    // the serial port or an error occurs.
    Read(p []byte) (n int, err error)

floitsch 09/18/2024 09:33 PM

I will google if others encountered the same issue.

RobvanLopik 09/18/2024 09:34 PM

I will call it a day, though here in Portugal it's an hour earlier than at your place

floitsch 09/18/2024 09:34 PM

for you it's 22:34 right?

floitsch 09/18/2024 09:35 PM

(23:34 here)

RobvanLopik 09/18/2024 09:35 PM

Exactly

floitsch 09/18/2024 09:35 PM

Could be this issue: https://github.com/bugst/go-serial/issues/148

Why read timeout return a nil error? · Issue #148 · bugst/go-serial

when i write a command, then i use io.ReadFull to read an response. But if the device do not response, it will STW. whether to return a read timeout error?

RobvanLopik 09/18/2024 09:35 PM

Why would monitor read the port?

floitsch 09/18/2024 09:36 PM

jag monitor

reads the serial port. That's where the messages come from.

RobvanLopik 09/18/2024 09:36 PM

Sorry, I was thinking from the ESP point of view

floitsch 09/18/2024 09:37 PM

I will leave you now.
If you have new information (like different behavior on Linux) let us know.

Also: I don't think we really solved anything for you yet. Make sure we get your attention if nothing happens.

RobvanLopik 09/18/2024 09:38 PM

Thanks, good night

48 messages in total