Two ESP devices that are seemingly the same acting different when flashing `Invalid image block`

Toit

#Two ESP devices that are seemingly the same acting different when flashing `Invalid image block`

Thread channel in help

addshore 10/09/2024 10:39 AM

We have 2 ESPc3s that are seemiingly the same, however post flashing is behaving differently.
The bad one ultimately ends up with Invalid image block, can't boot.
The primary thing we notice is that the loads end up with different positions and lengths, so something may be wrong with the bootloader and or partition table, but not sure how this happened (if we caused it)? and how to resolve.

Bad

ESP-ROM:esp32c3-api1-20210207
Build:Feb  7 2021
rst:0x1 (POWERON),boot:0xd (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd5820,len:0x48
load:0x00288200,len:0x20c
load:0x08a10862,len:0x20020aa
Invalid image block, can't boot.

And good

ESP-ROM:esp32c3-api1-20210207
Build:Feb  7 2021
rst:0x1 (POWERON),boot:0xd (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd5820,len:0x48
load:0x403cc710,len:0x65c
load:0x403ce710,len:0x2480
entry 0x403cc710
[toit] INFO: starting <v2.0.0-alpha.152>
[toit] DEBUG: clearing RTC memory: invalid checksum
[toit] INFO: running on ESP32C3 - revision 0.4

addshore 10/09/2024 10:40 AM

TLDR; any pointers would be greatly appreciated, as we can't currently identify what makes sense to look at next

bitphlipphar 10/09/2024 10:43 AM

Do you expect the bits in the flash to be the same on the two devices?

bitphlipphar 10/09/2024 10:43 AM

You can use esptool to read them from both devices and compare.

bitphlipphar 10/09/2024 10:44 AM

How did you flash them in the first place?

addshore 10/09/2024 10:44 AM

jag flash -c ESP32c3

addshore 10/09/2024 10:46 AM

So I think yes we would essentially expect everything to to be and behave the same on both at flash and boot time

addshore 10/09/2024 10:47 AM

This was chris runing this as he says

has the jag esp32c3 image changed since last week? It's probably not that.. but any suggestions? afaik we havent changed anything and the issue has been replicated on 2 new boards

I have taken a look at things and also can't really figure out the direction to investigate in

addshore 10/09/2024 10:53 AM

The flash output looks the same (left works, right doesnt boot) https://www.diffchecker.com/jl2hEJbN/

addshore 10/09/2024 10:56 AM

So the difference we see is on boot with a second load of load:0x403cc710,len:0x65c working, and a second load of load:0x00288200,len:0x20c failing, and trying to determine why they are each loading from different places for different lengths given the same flashing operation

bitphlipphar 10/09/2024 10:57 AM

Have you tried fetching the flash bits using esptool?

addshore 10/09/2024 10:59 AM

I'll get chris to have a look at that once he gets out of his meeting, its all currently setup on his desk and these board have a fair few wires attached

bitphlipphar 10/09/2024 11:00 AM

addshore 10/09/2024 11:00 AM

https://docs.espressif.com/projects/esptool/en/latest/esp32/esptool/basic-commands.html#read-flash-contents-read-flash looks like the thing I need?

bitphlipphar 10/09/2024 11:01 AM

Yup!

addshore 10/09/2024 11:30 AM

so, indeed there is some difference

addshore 10/09/2024 11:31 AM

does jag flash the bootloader too?
I noticed that the suggested flash command once building a custom envelope does for example

bitphlipphar 10/09/2024 11:34 AM

Yeah, it should flash the bootloader.

bitphlipphar 10/09/2024 11:35 AM

The envelope is an ar file, so you can use standard tooling to find the bootloader.bin entry.

bitphlipphar 10/09/2024 11:37 AM

ar -t ~/.cache/jaguar/v2.0.0-alpha.161/envelopes/firmware-esp32.envelope 
$envelope
$metadata
$firmware.bin
system
$properties
$firmware.elf
$bootloader.bin
$partitions.bin
$partitions.csv
$otadata.bin
$flashing.json

addshore 10/09/2024 11:37 AM

I guess my custom envelope is probably irellevant to this, as this happens with the default toit espc3 envelope too

bitphlipphar 10/09/2024 11:38 AM

So can you reproduce this after reflashing with the esp32c3 envelope pulled from https://github.com/toitlang/envelopes/releases/download/v2.0.0-alpha.163/firmware-esp32c3.envelope.gz ?

bitphlipphar 10/09/2024 11:38 AM

(or whatever version is associated with the version of jaguar you use)

bitphlipphar 10/09/2024 11:39 AM

jag version

should tell you.(edited)

addshore 10/09/2024 11:41 AM

So ignore what I said about custom evelopes I think, so far this has all been using jag flash -c ESP32c3 on jag v1.43.1 v2.0.0-alpha.163 and jag v1.41 alpah 160

addshore 10/09/2024 11:41 AM

and I assume that has fetched the envelopes from the toitlang repos?

addshore 10/09/2024 11:44 AM

If so, then the answer to can you reproduce this after reflashing with the esp32c3 envelope pulled from would be yes

addshore 10/09/2024 11:45 AM

Chris just said "oooh i have an idea" and left the room

bitphlipphar 10/09/2024 11:48 AM

Are you using that capitalization of ESP32c3 when you flash?

bitphlipphar

Are you using that capitalization of ESP32c3 when you flash?

addshore 10/09/2024 11:48 AM

nope

addshore 10/09/2024 11:48 AM

Chris has walked back in and it works! It was a hardware issue

addshore 10/09/2024 11:49 AM

Something in the board (that we are not using) was seemingly (potentailly) pulling a pin in a direction that caused all of this, so chopping it out for now has resolved out issue

addshore 10/09/2024 11:50 AM

Thanks for the guidance through trying to figure it out!

bitphlipphar 10/09/2024 11:50 AM

Freaky

addshore 10/09/2024 11:55 AM

yeah, took a chip off and its working, got to love prototype hardware

35 messages in total