The amount of SD card files slows down Ota

Toit

#The amount of SD card files slows down Ota

Thread channel in help

Rikke 03/08/2023 08:14 AM

Hello,

I noticed some varying Ota speeds on our toit devices, and decided to investigate the problem. The speeds would vary from 40 seconds to over 3 minutes. After i cleaned the SD card on the slow devices, it brought the time down to like 40 seconds.

I made some tests with the SD-card, by making big files and many files to see the impact on the Ota speed.
The files I made are in a folder being used by the device. I tried putting the files outside the working folder, but that did not change the Ota speed.

In conclusion: The amount of files present in a working folder slows down the device.

I have trouble finding out why this is the case.
- We are appending to a single file 0-3 times a second
- SPI clock is at 400 kHz
- The host package is used to create a file stream to the 1 file
- The Ota data is received with http client
- Ota data is written with firmware.FirmwareWriter writer

Rikke 03/08/2023 08:16 AM

The time recorded in the table is from sending the Ota message, till the new version is verified on mqtt

bitphlipphar 03/08/2023 08:50 AM

That is very interesting. Is it possible to try to get some memory stats before the OTA process starts? I'm curious if it is slower because of memory pressure. Are you running everything from within the same container?

Rikke 03/08/2023 08:51 AM

I can try and get some memory stats, and yes everything is from the same container

bitphlipphar 03/08/2023 09:11 AM

If it is from one container, then both print_objects and serial_print_heap_report can be useful.

Rikke 03/08/2023 09:13 AM

I will try print_objects

had to get some things ready

Rikke 03/08/2023 09:18 AM

Im gonna log allocated memory and system free memory with process stats, looks like print_objects prints on serial

Rikke 03/08/2023 09:20 AM

This sd card only has a few files:

(2023-03-08 09:18:59) Alloc 61212 | free 41096 before ota starts
(2023-03-08 09:19:23) Alloc 76872 | free 16288 when its finished

Rikke 03/08/2023 09:26 AM

If im printing [STATS_INDEX_ALLOCATED_MEMORY] [STATS_INDEX_SYSTEM_FREE_MEMORY][STATS_INDEX_BYTES_ALLOCATED_IN_OBJECT_HEAP]
(2023-03-08 09:23:12) Alloc 57212 | free 46704 | heap 228268 before
(2023-03-08 09:23:37) Alloc 57224 | free 44240 | heap 405264 after

only heap is really different this time. i will see if i can make print_objects return instead of sending on serial

bitphlipphar 03/08/2023 09:36 AM

You may want to use --gc for the process_stats call.

Rikke 03/08/2023 09:40 AM

--gc

is not in alpha 47

Rikke 03/08/2023 09:40 AM

But i should be able to add it

Rikke 03/08/2023 09:42 AM

hmm no, other changes has been made to process stats

Rikke 03/08/2023 09:56 AM

Okay returning the encoded_histogram in print_objects did not go well. It restarts the esp

Rikke 03/08/2023 09:58 AM

I am able to get serial output tomorrow, so I will just park it here for now

MikkelD 03/08/2023 10:51 AM

My thoughts. The host package is synchroneous, so it will block during file io and also limit the http <-> firmware.FirmwareWriter task.
You can verify this by adding timestamps to the ota loop to see if it get called less frequently when there are many files on the sdcard.

Rikke 03/08/2023 10:53 AM

Thank you for your input, I will test it tomorrow !

MikkelD

My thoughts. The host package is synchroneous, so it will block during file io and also limit the http <-> firmware.FirmwareWriter task. You can verify this by adding timestamps to...

Rikke 03/09/2023 08:39 AM

Okay I tried this and you are right, it does get called less frequently. it is doing less writes to the firmware writer

erikcorry_arbat 03/09/2023 08:42 AM

The host package was originally developed on Linux which does not have support for getting an event from the OS when file operations have completed. Linux will use huge disk caches that fill all available memory, so this is not normally a huge issue. It might be a bigger issue on file operations on ESP32.

Rikke 03/09/2023 08:51 AM

250 files on the card is also unlikely to happen for us, but I was just curious and wanted to understand why

I guess I have to make sure we don't create too many files.

Oliver M 03/09/2023 09:04 AM

Will having subdirectories help?

Rikke 03/09/2023 09:06 AM

I didnt try with subdirectories in the current folder, but that solution would cost more to implement, than reducing the amount of files generated

MikkelD 03/09/2023 10:59 AM

What would help you (here and now) is to use the microservice framework and then "spawn" the service responsible for either FW-update or SD-card writes. This makes it run in a separate FreeRTOS task on the ESP32 and the host package blocking will not affect the other process. I am doing something very similar to avoid TLS decryption to interfere with the audio data path.

Rikke 03/09/2023 11:05 AM

Thats a great idea! I will try that after lunch

Rikke 03/09/2023 12:24 PM

Okay we have tried putting the ota in spwn before, but we have problems with the http connection. But i will see if i can make it work anyway

Rikke 03/09/2023 12:44 PM

Oh wow, this reduced the time from 258s to 75s

thank you so much

MikkelD 03/09/2023 12:48 PM

You are welcome. The SD-card lib should probably not block though

It is now on my list.

27 messages in total