I'm getting an OUT OF MEMORY error when using MQTT connecting to Azure IoT Hub

Toit

#I'm getting an OUT OF MEMORY error when using MQTT connecting to Azure IoT Hub

Thread channel in help

PeterJ 05/27/2023 09:40 AM

When the message payload have been delivered to the IoT Hub and I close the connection using client.close, I get the following OUT OF MEMOTY error constantly:

OUT OF MEMORY error.
0: out_ofmemory <sdk>/core/exceptions.toit:195:1
1: Map.remove <sdk>/core/collections.toit:2474:17
2: HashedInsertionOrderedCollection.remove <sdk>/core/collections.toit:1936:5
3: RpcSynchronizer.send.<block> <sdk>/rpc/rpc.toit:66:11
4: Monitor.locked_.<block> <sdk>/core/monitorimpl.toit:123:12
5: Monitor.locked <sdk>/core/monitorimpl.toit:95:3
6: RpcSynchronizer.send <sdk>/rpc/rpc.toit:53:3
7: Rpc.invoke.<block> <sdk>/rpc/rpc.toit:30:21
8: Rpc.invoke <sdk>/rpc/rpc.toit:32:52
9: invoke <sdk>/rpc/rpc.toit:8:23
10: TcpSocketImpl_.write <sdk>/net/impl.toit:132:15
11: Session.flushoutgoing <sdk>/tls/session.toit:199:25
12: Session.write <sdk>/tls/session.toit:145:9
...
28: Connection_.write <pkg:mqtt>/fullclient.toit:153:14
29: FullClient.disconnect.<block>.<block> <pkg:mqtt>/fullclient.toit:907:23
30: Mutex.do.<block> <sdk>/monitor.toit:24:27
31: Monitor.locked.<block> <sdk>/core/monitorimpl.toit:123:12
32: Monitor.locked <sdk>/core/monitorimpl.toit:95:3
33: Mutex.do <sdk>/monitor.toit:24:3
34: FullClient.disconnect.<block> <pkg:mqtt>/fullclient.toit:905:19
35: Mutex.do.<block> <sdk>/monitor.toit:24:27
36: Monitor.locked.<block> <sdk>/core/monitorimpl.toit:123:12
37: Monitor.locked <sdk>/core/monitorimpl.toit:95:3
38: Mutex.do <sdk>/monitor.toit:24:3
39: FullClient.disconnect <pkg:mqtt>/full_client.toit:899:14
40: FullClient.close <pkg:mqtt>/full_client.toit:883:19
41: Client.close <pkg:mqtt>/client.toit:214:13

PeterJ 05/27/2023 09:43 AM

Also getting this error and a subsequent reset: Reset after exiting deep sleep mode. crashes=2 out-of-memory=1

floitsch 05/27/2023 09:43 AM

It seems to happen during the disconnect.

PeterJ 05/27/2023 09:43 AM

Yes, I think so

floitsch 05/27/2023 09:44 AM

A clean close sends a disconnect packet which seems to lead to this situation.
Could you maybe try a forced close, just to see if that changes anything?

floitsch 05/27/2023 09:45 AM

close --force

floitsch 05/27/2023 09:45 AM

In that case it doesn't notify the other side of the closing down.

PeterJ 05/27/2023 09:45 AM

Checking

PeterJ 05/27/2023 09:47 AM

That did the trick. Just slam the door shut

floitsch 05/27/2023 09:47 AM

Clearly not the right solution, but hopefully unblocks you.

floitsch 05/27/2023 09:48 AM

Could you maybe tag the current broken one, so we can investigate?
If you could send us a repro that would be ideal.

PeterJ 05/27/2023 09:49 AM

Not the nicest way to do it, but it works. I will send you the code and the deviceId so you can have a look at it if you want?

floitsch 05/27/2023 09:49 AM

That would be great!

floitsch 05/27/2023 09:49 AM

Basically anything that allows me to reproduce it locally

floitsch 05/27/2023 09:50 AM

I don't think I will have the time over the weekend, but early next week.

PeterJ 05/27/2023 09:52 AM

It's in your inbox now

PeterJ 05/27/2023 09:53 AM

Thanks mate

bitphlipphar 05/27/2023 10:40 AM

This might work considerably better on Toit v2.

bitphlipphar 05/27/2023 10:40 AM

We have optimized the memory consumption of TLS connections a lot.

bitphlipphar

This might work considerably better on Toit v2.

floitsch 05/27/2023 10:58 AM

I was already thinking the same thing. Might be worth trying on v2 first before I spend time debugging.

floitsch 05/27/2023 11:00 AM

Another thing to try might be delaying the close a bit. Might be that there is already data queued and the disconnect request is just too much. By waiting the queued data could be offloaded and the disconnect would then work.

PeterJ 05/27/2023 11:16 AM

Can I migrate to version 2 by now?

floitsch 05/27/2023 11:19 AM

I think it's a good time to start migrating.

PeterJ 05/27/2023 11:21 AM

Okay, then we just need to plan it so I can get started on it

24 messages in total